Image capture apparatus and method for generating combined-image data

ABSTRACT

An image capture apparatus includes an image capture unit configured to capture an image of a subject to obtain image data, an image-capture control unit configured to allow the image capture unit to execute an image capture operation of capturing a plurality of frames of image data having continuity in time so that a frame rate of the image capture unit is changed in accordance with a change of a subject brightness level during the execution of the image capture operation, and a combination processing unit configured to perform a combination process using, as combination-use image data to be combined, the plurality of frames of image data having continuity in time captured by the image capture unit so as to generate combined-image data representing a still image.

CROSS REFERENCES TO RELATED APPLICATIONS

The present application claims priority from Japanese Patent ApplicationNo. JP 2008-078017, filed in the Japanese Patent Office on Mar. 25,2008, the entire content of which is incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image capture apparatus and method.

2. Description of the Related Art

Japanese Unexamined Patent Application Publications No. 2005-354166, No.2004-219765 (corresponding to U.S. Pat. No. 7,295,232B2), No. 2006-86933(corresponding to US 2006062433A1), and No. 2006-174069 (correspondingto US 2006127084A1) are examples of related art.

A long-time exposure technique is common as a photographic or imagecapture technique. In this technique, exposure is continuously performedfor a certain period of time such as several seconds to several tens ofseconds or even over several tens of minutes.

The long-time exposure technique is also used to provide photographicrepresentations as well as to adjust the brightness of a subject.

For example, night scenes are captured with a long exposure time. Sincethe amount of light is low, the exposure time is increased to collect asufficient amount of light to obtain shots of night scenes.

The long-time exposure technique may also be used for other purposes.For example, the aperture is intentionally set low or the image capturesensitivity is reduced to represent the motion of a subject or focus ona stationary object among moving objects.

Image capture techniques such as using a flash (flashlight device)during long-time exposure to achieve various effects such asfirst-curtain synchronization, second-curtain synchronization, andmulti-flash are also common.

SUMMARY OF THE INVENTION

However, it is difficult for users to obtain their desired images usinglong-time exposure image capture.

Some recent image capture apparatuses (such as digital still cameras)are designed such that the cameras perform metering in the normal imagecapture process to determine appropriate exposure settings (aperture andshutter speed). However, in dark conditions involving long-timeexposure, the light level is outside the metering range determined bythe cameras and the exposure settings are not provided. In such cases,it is necessary for users to determine settings such as the exposuretime and the shutter time according to their experience and intuition inorder to perform long-time exposure image capture. In practice, this isdifficult for inexperienced users.

Japanese Unexamined Patent Application Publication No. 2005-354166discloses a technique for achieving the image effect of long-timeexposure using a combination process.

This technique includes real-time observation of an image which is beingproduced as a result of light received by solid-state image captureelements during exposure, selection and combination of a plurality ofimages after image capture that are divisionally generated duringexposure, and removal of an undesired image during exposure.

This technique provides a favorable image using long-time exposurephotography under image capture conditions where the subject is notmoving (there is little motion or, even though the subject is moving, nomotion blur occurs because the exposure time is sufficiently long). Thisis effective to capture non-motion scenes such as night scenes using along exposure time.

However, it is difficult to obtain a favorable image when a user wishesto capture a still image of a moving subject, for example, an imagerepresenting the motion of a subject or an image in which a stationaryobject is focused on among moving objects.

It is also difficult to obtain a still image that provides a smoothrepresentation of motion by removing an undesired image during exposure(such as an image of a night scene which is captured when surroundingsare illuminated by the headlight of a vehicle passing by).

While the first-curtain synchronization, second-curtain synchronization,and multi-flash effects described above are used as photographic orimage capture representation techniques, it is necessary for users todetermine settings such as the shutter time, the exposure time, and theamount of flash illumination according to their experience and intuitionin order to obtain their desired images. This is also difficult forinexperienced users.

Since there are actually “flash prohibited” environments in publicspaces such as museums, images with effects such as first-curtainsynchronization, second-curtain synchronization, and multi-flash may notnecessarily be obtained.

Furthermore, since firing a flash consumes power, a battery or capacitorfor a flash is necessary. This may inhibit the reduction in device size,power consumption, and cost.

Furthermore, it is often necessary to use a tripod to hold a cameraduring long-time exposure to prevent the camera from moving.Photographers or users therefore purchase accessories in addition to acamera body, such as a tripod and also carry the accessories. Thetransportation of the camera and accessories will be a burden for theusers to discourage the user from readily capturing images with a longexposure time.

During long-time exposure, further, movements of subjects, camera shake,or the like may cause blurring of all moving subjects in a capturedimage. Thus, it is difficult to obtain sufficient image captureconditions for capturing a desired image.

Moreover, during long-time exposure image capture, for example, thefining of a flash by another photographer or instantaneous illuminationfrom the headlight of a vehicle passing by would not provide a favorablecombined image.

It is therefore desirable to allow general users who are not experts toeasily achieve various image effects, in particular, an image effectsimilar to the long-time exposure effect or image effects achieved usinglong-time exposure such as first-curtain synchronization, second-curtainsynchronization, and multi-flash. It is also desirable to allow users toeasily obtain their desired images.

In an embodiment of the present invention, an image capture apparatusincludes an image capture unit configured to capture an image of asubject to obtain image data, an image-capture control unit configuredto allow the image capture unit to execute an image capture operation ofcapturing a plurality of frames of image data having continuity in timeso that a frame rate of the image capture unit is changed in accordancewith a change of a subject brightness level during the execution of theimage capture operation, and a combination processing unit configured toperform a combination process using, as combination-use image data to becombined, the plurality of frames of image data having continuity intime captured by the image capture unit so as to generate combined-imagedata representing a still image.

The image-capture control unit may perform exposure adjustment controlduring the execution of the image capture operation by controlling anaperture mechanism provided in an optical system of the image captureunit and by variably controlling the frame rate.

The image-capture control unit performs exposure adjustment controlduring the execution of the image capture operation by controlling alight intensity filter mechanism provided in an optical system of theimage capture unit and by variably controlling the frame rate.

The image-capture control unit may perform exposure adjustment controlduring the execution of the image capture operation by controlling avariable gain circuit provided in an image-capture signal processingsystem of the image capture unit in response to a captured image signaland by variably controlling the frame rate.

The image capture apparatus may further include a recording unitconfigured to record the plurality of frames of image data havingcontinuity in time captured by the image capture unit onto a recordingmedium as a sequence of image data used for a combination process.

The combination processing unit may perform a combination process usingthe plurality of frames of image data having continuity in time recordedonto the recording medium so as to generate combined-image datarepresenting a still image.

The image capture apparatus may further include an operation detectionunit configured to detect operation input information used for acombination process. The combination processing unit may perform acombination process on combination-use image data of frames in a rangeon a time axis specified by the operation input information among thecombination-use image data having continuity in time so as to generatecombined-image data representing a still image.

The image capture apparatus may further include an operation detectionunit configured to detect operation input information used for acombination process. The combination processing unit may perform acombination process on combination-use image data of each of a pluralityof frames using a weighting coefficient specified by the operation inputinformation so as to generate combined-image data representing a stillimage.

The combination processing unit may perform a combination process oncombination-use image data of each of a plurality of frames usingweighted averages so as to generate combined-image data representing astill image.

The combination processing unit may perform a combination process oncombination-use image data of each of a plurality of frames by assigninga weighting coefficient, the weighting coefficient being determined asan inverse of a ratio of lengths of frame periods of the combination-useimage data in the image capture operation, so as to generatecombined-image data representing a still image.

The combination processing unit may perform a combination process usingthe combination-use image data and interpolated image data, theinterpolated image data being generated by an interpolation processusing the combination-use image data, so as to generate combined-imagedata representing a still image.

The image capture apparatus may further include a display control unitconfigured to output the combined-image data generated by thecombination processing unit as image data used for display.

The recording unit may record the combined-image data generated by thecombination processing unit onto a recording medium.

The image capture apparatus may further include a sending unitconfigured to send the combined-image data generated by the combinationprocessing unit to an external device.

In another embodiment of the present invention, an image capture methodincludes the steps of executing an image capture operation of capturinga plurality of frames of image data having continuity in time, changinga frame rate of image capture in accordance with a change of a subjectbrightness level during the execution of the image capture operation,and performing a combination process using, as combination-use imagedata to be combined, the plurality of frames of image data havingcontinuity in time so as to generate combined-image data representing astill image.

According to the embodiments of the present invention, during theexecution of capturing a plurality of frames of image data havingcontinuity in time, a frame rate of an image capture unit is changed inaccordance with a change in subject brightness. That is, in an exposureadjustment method, the brightness level of a captured image is adjustedby changing the frame rate so that a plurality of frames of image datahaving continuity in time can have a constant brightness level to someextent. In a case where the amount of exposure is adjusted by changingthe frame rate, it is not necessary to change the exposure time within aperiod of one frame using the so-called electronic shutter. This canminimize lack of information regarding a subject image.

This is suitable for obtaining a combined image with a long-timeexposure effect by combining a plurality of frames of image data havingcontinuity in time, or a sequence of image data captured in a moviefashion. In order to obtain a smooth representation of an image effectas a long-time exposure image, desirably, brightness levels (forexample, average brightness) of images are constant and there is no lackof information on the time axis.

In this image capture apparatus, a sequence of image data obtained bythe image capture operation is used as combination-use image data to becombined, and a combination process is performed. In this case, a rangeof images to be combined (a range on the time axis) or weightingcoefficients to be assigned to the images are set in accordance with anoperation. Thus, image combination can be realized according to theuser's intention. That is, after capturing images, a user selects framesto perform image combination, thus easily achieving an image effectsimilar to the effect of an image captured using long-time exposure. Inaddition, by applying weighting to each of frames to be combined,effects such as first-curtain synchronization, second-curtainsynchronization, and multi-flash can be achieved.

According to the embodiments of the present invention, a frame rate ischanged during image capture in accordance with a change in subjectbrightness of a sequence of image data used for a combination processfor achieving, for example, a long-time exposure effect to adjust thebrightness of a captured image. This ensures that a plurality of framesof images having continuity in time that provide constant subjectbrightness levels of the images and that minimize lack of informationregarding subject images can be obtained. By combining such a sequenceof image data, a special effect image that provides a smoothrepresentation of the motion of a subject can be obtained.

According to the embodiments of the present invention, furthermore,image effects similar to long-time exposure and other special effects,which may be achievable only by experts in the related art, and imageeffects that could not have been achieved in image capture of therelated art can be easily achieved by general users. For example,enhanced photographic representations or more creative photographicrepresentations can be promoted. In addition, improved image quality canalso be achieved.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an image capture apparatus according to anembodiment of the present invention;

FIGS. 2A and 2B are external front and rear views of the image captureapparatus according to the embodiment, respectively;

FIG. 3 is a block diagram showing a functional structure of a centralprocessing unit (CPU) in the image capture apparatus according to theembodiment;

FIG. 4 is a diagram showing a mode operation of the image captureapparatus according to the embodiment;

FIG. 5 is a flowchart showing a camera-mode process of the image captureapparatus according to the embodiment;

FIG. 6 is a diagram showing an image captured using the image captureapparatus according to the embodiment;

FIG. 7 is a flowchart showing-a combining preparatory process accordingto the embodiment;

FIG. 8 is a flowchart showing a combination process according to theembodiment;

FIG. 9 is a flowchart showing an adjustment process according to theembodiment;

FIG. 10 is a diagram showing a combination-work image obtained at thebeginning of playback according to the embodiment;

FIG. 11 is a diagram showing a combination-work image obtained when acombination start position is specified according to the embodiment;

FIG. 12 is a diagram showing a combination-work image obtained when acombination end position is specified according to the embodiment;

FIG. 13 is a diagram showing a combination-work image (with a long-timeexposure effect) obtained at an initial state in an adjustment processaccording to the embodiment;

FIG. 14 is a diagram showing a combination-work image obtained whenweighting coefficients are changed so as to achieve the first-curtainsynchronization effect according to the embodiment;

FIG. 15 is a diagram showing a combination-work image obtained when acombination range is changed in the state shown in FIG. 14 according tothe embodiment;

FIG. 16 is a diagram showing a combination-work image obtained whenweighting coefficients are changed so as to achieve the second-curtainsynchronization effect the according to the embodiment;

FIG. 17 is a diagram showing a combination-work image obtained when acombination range is changed in the state shown in FIG. 16 according tothe embodiment;

FIG. 18 is a diagram showing a combination-work image obtained whenweighting coefficients are changed so as to achieve the multi-flasheffect according to the embodiment;

FIG. 19 is a flowchart showing an exemplary process for displaying acombined image before the change in an adjustment process according tothe embodiment;

FIG. 20 is a diagram showing a combination-work image obtained when acombined image before the change is displayed in the adjustment processaccording to the embodiment;

FIG. 21 is a diagram showing a coefficient template selection screenaccording to the embodiment;

FIG. 22 is a flowchart showing an exemplary process using coefficienttemplates according to the embodiment;

FIG. 23 is a diagram showing a coefficient template selection screenduring image capture according to the embodiment;

FIGS. 24A to 24D are diagrams showing combined images obtained with andwithout using an electronic shutter according to the embodiment;

FIGS. 25A to 25D are diagrams showing examples of exposure adjustmentcontrol methods according to the embodiment;

FIGS. 26A to 26D are diagrams showing exposure adjustment controlmethods using preferentially functions other than the electronic shutteraccording to the embodiment;

FIG. 27 is a flowchart showing exposure adjustment control usingpreferentially functions other than the electronic shutter according tothe embodiment;

FIGS. 28A to 28D are diagrams showing combined images obtained usingdivisional exposure at a fixed frame rate according to the embodiment;

FIGS. 29A to 29C are diagrams showing the generation of frames usingdivisional exposure at a fixed frame rate according to the embodiment;

FIGS. 30A to 30D are diagrams showing combined images obtained usingcontinuous and discontinuous exposure times according to the embodiment;

FIGS. 31A to 31C are diagrams showing a variable-frame-rate operationaccording to the embodiment;

FIGS. 32A to 32D are diagrams showing exposure adjustment methods usingvariable frame rates according to the embodiment;

FIG. 33 is a flowchart showing exposure adjustment control usingvariable frame rates according to the embodiment;

FIGS. 34A to 34D are diagrams showing combined images obtained with andwithout using inter-frame interpolation according to the embodiment;

FIGS. 35A and 35B are diagrams showing inter-frame interpolationaccording to the embodiment;

FIG. 36 is a flowchart showing an exemplary process including frameinterpolation according to the embodiment;

FIG. 37 is a diagram showing a combination-work image obtained whenflash removal is performed according to the embodiment;

FIG. 38 is a flowchart showing an exemplary process including flashremoval according to the embodiment;

FIG. 39 is a diagram showing a combination-work image obtained whenflash correction is performed according to the embodiment;

FIG. 40 is a flowchart showing an exemplary process including flashcorrection according to the embodiment;

FIG. 41 is a diagram showing a combination-work image according to theembodiment;

FIG. 42 is a diagram showing a combination-work image obtained when themulti-flash effect is achieved according to the embodiment;

FIG. 43 is a diagram showing a combination-work image obtained whendistance-based flash correction is performed according to theembodiment;

FIG. 44 is a flowchart showing an exemplary process includingdistance-based flash correction according to the embodiment;

FIG. 45 is a diagram showing a combination-work image obtained whendistance-based correction is performed for all images according to theembodiment;

FIG. 46 is a diagram showing a combination-work image obtained whenflash images are combined according to the embodiment;

FIG. 47 is a diagram showing another combination-work image obtainedwhen distance-based flash correction is performed according to theembodiment;

FIGS. 48A to 48C are diagrams showing combined images in which blurringoccurs according to the embodiment;

FIG. 49 is a diagram showing a combination-work image in which acombined image affected by camera shake is displayed according to theembodiment;

FIG. 50 is a diagram showing a combination-work image in which acombined image after camera-shake correction is displayed according tothe embodiment;

FIG. 51 is a diagram showing a combination-work image in which acombined image to which the multi-flash effect is applied aftercamera-shake correction is displayed according to the embodiment;

FIG. 52 is a flowchart showing an exemplary process includingcamera-shake correction according to the embodiment;

FIG. 53 is a diagram showing a combination-work image in which acombined image affected by subject blur is displayed according to theembodiment;

FIG. 54 is a flowchart showing an exemplary process includingsubject-blur correction according to the embodiment;

FIG. 55 is a flowchart showing an exemplary process includingcamera-shake correction and subject-blur correction according to theembodiment;

FIG. 56 is a flowchart showing another exemplary process includingcamera-shake correction and subject-blur correction according to theembodiment; and

FIG. 57 is a schematic diagram showing an example structure of aninformation processing apparatus according to an embodiment of thepresent invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

An embodiment of the present invention will be described hereinafter inthe following order:

-   1. Structure of Image Capture Apparatus-   2. Operation Mode-   3. Camera-Mode Processes-   4. Combination-Mode Process-   4-1: Combining Preparatory Process-   4-2: Combination Process-   4-3: Exemplary Adjustment Process Using Displayed Images before and    after Change-   5. Template-based Process-   6. Image Capture Operation at Fixed Frame Rate-   7. Image Capture Operation at Variable Frame Rate-   8. Exemplary Combination-Mode Process: Frame Interpolation-   9. Exemplary Combination-Mode Processes: Flash Removal/Correction-   10. Exemplary Combination-Mode Processes: Distance-based Correction-   11. Exemplary Combination-Mode Processes: Blurring Correction-   12. Information Processing Apparatus    1. Structure of Image Capture Apparatus

The structure of an image capture apparatus according to an embodimentof the present invention will now be described in the context of thestructure of a digital still camera with reference to FIGS. 1 to 3.

FIGS. 2A and 2B are external front and rear views of an image captureapparatus 1 according to an embodiment of the present invention,respectively. As shown in FIGS. 2A and 2B, the image capture apparatus 1may be, for example, a digital still camera which a general user who isnot an expert usually uses.

The image capture apparatus 1 includes an image capture lens unit 21 aand a flash light emitting unit 15 on a front side thereof, and adisplay panel 6 on a rear side thereof. The display panel 6 may be aliquid crystal panel, an organic electroluminescent (EL) panel, or thelike. The image capture apparatus 1 further includes operators atappropriate locations which are used for user operations. For example,operation keys 5 a, 5 b, 5 c, 5 d, 5 f, and 5 g serve as keys forproviding various operation functions, including a shutter operationkey, a mode operation key, a wide-angle/telephoto operation key, a menuoperation key, an exposure correction instruction key, and a playbackkey. Other operators including a dial operation unit 5 h and a cross key5 i are also disposed. The dial operation unit 5 h is used for selectionor the like of, for example, an image capture mode. The cross key 5 i isused for various operations such as selection/setting of an operationmenu item displayed on the display panel 6.

An example structure of the image capture apparatus 1 will be describedwith reference to, for example, FIG. 1.

As shown in FIG. 1, the image capture apparatus 1 includes an imagecapture system 2, a control system 3, a camera digital signal processor(DSP) 4, an operation unit 5, the display panel 6, a display controller7, an external interface (I/F) 8, a synchronous dynamic random accessmemory (SDRAM) 9, and a media interface 10.

The image capture system 2 is configured to execute an image captureoperation. The image capture system 2 includes a lens mechanism unit 21,an aperture/neutral density (ND) filter mechanism 22, an image captureelement unit 23, an analog signal processing unit 24, ananalog-to-digital (A/D) conversion unit 25, a lens driving unit 26, alens position detection unit 27, a timing generation circuit 28, a blurdetection unit 13, a light emission driving unit 14, the flash lightemitting unit 15, a lens driver 17, an aperture/ND driver 18, and animage capture element driver 19.

Incident light from a subject is directed into the image capture elementunit 23 through the lens mechanism unit 21 and the aperture/ND filtermechanism 22.

The lens mechanism unit 21 is incorporated in the image capture lensunit 21 shown in FIG. 2A, and has a plurality of optical lensesincluding a cover lens, a focus lens, and a zoom lens. The lens drivingunit 26 serves as a lens shifting mechanism for shifting the focus lensor zoom lens along an optical axis. When drive power is applied by usingthe lens driver 17, the lens driving unit 26 shifts the focus lens orzoom lens. The lens driver 17 is controlled by a central processing unit(CPU) 31, which will be described below, to execute focus control orzoom operations.

The aperture/ND filter mechanism 22 includes an aperture mechanism andan ND filter mechanism that is inserted into a lens optical system toattenuate (adjust) the amount of incident light. The aperture/ND filtermechanism 22 is configured to adjust the light intensity.

The aperture/ND driver 18 adjusts the amount of incident light byopening and closing the aperture mechanism. The aperture/ND driver 18also adjusts the amount of incident light by inserting and removing anND filter along the optical axis of the incident light. The CPU 31controls the aperture/ND driver 18 to drive the aperture mechanism orthe ND filter to control the amount of incident light (or performexposure adjustment control).

The light flux coming from the subject is transmitted through the lensmechanism unit 21 and the aperture/ND filter mechanism 22, and a subjectimage is formed on the image capture element unit 23.

The image capture element unit 23 photoelectrically converts the formedsubject image, and outputs a captured image signal corresponding to thesubject image.

The image capture element unit 23 has a rectangular image-capture areaformed of a plurality of pixels, and sequentially outputs image signals,each of which is an analog signal corresponding to an amount of electriccharge accumulated in one of the pixels, to the analog signal processingunit 24 on a pixel-by-pixel basis. The image capture element unit 23 maybe implemented by, for example, a charge coupled device (CCD) sensorarray, a complementary metal oxide semiconductor (CMOS) sensor array, orthe like.

The analog signal processing unit 24 includes internal circuits such asa correlated double sampling (CDS) circuit and an automatic gain control(AGC) circuit. The analog signal processing unit 24 performs apredetermined analog process on the image signal input from the imagecapture element unit 23.

The A/D conversion unit 25 converts the analog signal processed by theanalog signal processing unit 24 into a digital signal, and supplies thedigital signal to the camera DSP 4.

The timing generation circuit 28 is controlled by the CPU 31 to controlthe timings of the operations of the image capture element unit 23, theanalog signal processing unit 24, and the A/D conversion unit 25.

Specifically, the timing generation circuit 28 supplies signals forcontrolling the timing of the image capture operation of the imagecapture element unit 23 to the image capture element unit 23 through theimage capture element driver 19, such as anexposure/electric-charge-read timing signal, a timing signal forproviding an electronic shutter function, a transfer clock signal, and asynchronization signal according to a frame rate. The timing generationcircuit 28 also supplies the timing signals to the analog signalprocessing unit 24 so that the analog signal processing unit 24 canperform a process in synchronization with the transfer of an imagesignal from the image capture element unit 23.

The CPU 31 can control the timing signals generated by the timinggeneration circuit 28 to change the frame rate for image capture orperform electronic shutter control (intra-frame variable control ofexposure time). Further, for example, the CPU 31 can apply a gaincontrol signal to the analog signal processing unit 24 through thetiming generation circuit 28 to perform variable gain control of acaptured image signal.

The blur detection unit 13 is configured to detect the amount of camerashake. The blur detection unit 13 is formed of, for example, anacceleration sensor, a vibration sensor, or the like, and supplies thedetected information to the CPU 31 as the amount of blur.

The flash light emitting unit 15 is driven by the light emission drivingunit 14 to emit light. The CPU 31 instructs the light emission drivingunit 14 to emit flash light at a predetermined time specified in a useroperation or the like so that light can be emitted from the flash lightemitting unit 15.

The camera DSP 4 performs various digital signal processes on thecaptured image signal input from the A/D conversion unit 25 of the imagecapture system 2.

In the camera DSP 4, for example, as shown in FIG. 1, processingfunctions such as an image signal processing unit 41, acompression/decompression processing unit 42, an SDRAM controller 43,and an information generation unit 44 are implemented by internalhardware or software.

The image signal processing unit 41 performs a process on the inputcaptured image signal. For example, the image signal processing unit 41performs arithmetic processing for controlling the driving of the imagecapture system 2 using the captured image signal, such as autofocus (AF)processing and auto-iris (automatic exposure (AE)) processing, and alsoperforms processing for the input captured image signal itself, such asautomatic white balance (AWB) processing.

For example, in the autofocus processing, the image signal processingunit 41 performs contrast detection of the input captured image signal,and sends the detected information to the CPU 31. Various controltechniques are available as autofocus control methods. In a techniquecalled contrast AF, contrast detection of the captured image signal isperformed at each time point with the focus lens forcibly moved, and aposition of the focus lens in an optimum contrast state is determined.Specifically, prior to the image capture operation, the CPU 31 performscontrol so as to check the contrast detection value detected by theimage signal processing unit 41 while controlling the movement of thefocus lens and to set a position at an optimum contrast state as anoptimum focus position.

During image capture, the CPU 31 can perform focus control using adetection method called wobbling AF. During the image capture operation,the CPU 31 checks the contrast detection value detected by the imagesignal processing unit 41 while causing the focus lens to slightly moveback and forth constantly. Although the optimum position of the focuslens may vary depending on the situation of the subject, contrastdetection is performed by slightly displacing the focus lens back andforth, thereby determining changes in a focus control direction inaccordance with changes of the subject. Accordingly, autofocus can beexecuted in accordance with subject conditions.

Note that the lens shifting mechanism in the lens driving unit 26 isassigned addresses for individual shift positions, and a lens positionis identified using the addresses of the shift positions.

The lens position detection unit 27 identifies the address of thecurrent lens position of the focus lens to calculate the distance to anin-focus subject, and supplies distance information regarding thecalculated distance to the CPU 31. Therefore, the CPU 31 can determinethe distance to the main subject that is in focus.

In the auto-iris processing performed by the image signal processingunit 41 of the camera DSP 4, for example, the subject brightness iscalculated. For example, the average brightness of the input capturedimage signal is calculated and subject brightness information, orexposure information, regarding the calculated average brightness issupplied to the CPU 31. The average brightness can be calculated usingvarious methods such as calculating an average value of brightnesssignals of all pixels of one frame of captured image data or calculatingan average value of brightness signals when a weight is assigned to acenter portion of an image.

The CPU 31 can perform automatic exposure control based on the exposureinformation. Specifically, exposure adjustment is performed using theaperture mechanism, the ND filter, electronic shutter control in theimage capture element unit 23, or gain control for the analog signalprocessing unit 24.

The image signal processing unit 41 of the camera DSP 4 performs, inaddition to the process for generating the signals used for theautofocus operation and auto-iris operation, signal processes on thecaptured image signal itself such as automatic white balance, gamma (γ)correction, edge enhancement, and camera-shake correction.

The compression/decompression processing unit 42 in the camera DSP 4performs a compression process on the captured image signal or adecompression process on compressed image data. For example, thecompression/decompression processing unit 42 performs a compressionprocess/decompression process according to a technique such as a JointPhotographic Experts Group (JPEG) or Moving Picture Experts Group (MPEG)technique.

The SDRAM controller 43 performs writing/reading on the SDRAM 9. TheSDRAM 9 is used to, for example, temporarily store the captured imagesignal input from the image capture system 2, store data or reserve awork area in the process performed by the image signal processing unit41 or the compression/decompression processing unit 42, or storeinformation obtained by the information generation unit 44. The SDRAMcontroller 43 performs writing/reading of such data on the SDRAM 9.

The information generation unit 44 generates information used forvarious operations in a combination process described below. Forexample, the information generation unit 44 generates distancedistribution information indicating the distances to subjects in acaptured image signal screen. The distance distribution information maybe, for example, information regarding the distances to subjects inunits of pixels as well as the distance to the main subject. Theinformation is also called a depth map.

The determination of pixel-based distance information for generatingdistance distribution information can be executed by analyzing theamount of blur during the wobbling AF described above or the like.Alternatively, a light emitting unit (not shown) configured to emitauxiliary light having a specific wavelength of non-visible light may beprovided and a period of time during which the light of the specificwavelength returns after it has been emitted may be measured todetermine the distance to a subject on a pixel-by-pixel basis.

The control system 3 includes the CPU 31, a random access memory (RAM)32, a flash read-only memory (ROM) 33, and a clock circuit 34. Each unitin the control system 3, each unit in the camera DSP 4, each unit in theimage capture system 2, the display controller 7, the external interface8, and the media interface 10 are configured to communicate image dataor control information with one another via a system bus.

The CPU 31 controls the overall operation of the image capture apparatus1. Specifically, the CPU 31 performs various arithmetic processes orexchanges control signals or the like with the corresponding unitsaccording to a program stored in an internal ROM or the like andaccording to a user operation using the operation unit 5 to cause theunits to execute necessary operations. The CPU 31 also performs furtherprocesses for image combination described below such as arithmeticprocessing and image analysis processing.

The RAM 32 temporarily stores the captured image signal (image data ofeach frame) processed by the camera DSP 4, or stores image data used fora combination process described below and other informationcorresponding to various processes of the CPU 31.

The flash ROM 33 is used to store image data representing a capturedimage (which has been captured by a user as a still image or a movingimage) or other information to be saved in a non-volatile fashion. Theflash ROM 33 may also be used to store a software program forcontrolling the image capture apparatus 1, camera setting data, or thelike. The flash ROM 33 is also used to store coefficient templates usedfor a combination process described below.

The clock circuit 34 performs time counting to determine current timeinformation (year, month, day, hour, minute, and second).

The operation unit 5 includes the operators shown in FIGS. 2A and 2B anda signal generation unit for generating signals according to theoperations of the operators. User operation information based on theoperators is transmitted from the operation unit 5 to the CPU 31.

The operation unit 5 may be configured to allow touch panel operationsas well as operations using the operators. Specifically, the displaypanel 6 may be provided with a touch sensor so that an operation inputcan be performed in response to a touch of the screen by the user.

The display controller 7 causes the display panel 6 to execute anecessary display operation under the control of the CPU 31. Examples ofdisplay operations on the display panel 6 may include display of amonitor (so-called Live View Display or display of amoving-image/still-image capturing monitor), display of a playback imageread from the recording medium 90 or the flash ROM 33, display of anoperation menu, display of various icons, display of time and date, anddisplay regarding a combination process described below.

The media interface 10 performs reading/writing of data on the recordingmedium 90, such as a memory card (a card-shaped removable memory) placedin the image capture apparatus 1, under the control of the CPU 31. Forexample, the media interface 10 performs an operation of recordingstill-image data or moving-image data obtained as a result of imagecapture onto the recording medium 90. The media interface 10 furtherperforms an operation of reading image data used for a combinationprocess described below from the recording medium 90.

While the recording medium 90 is implemented as a portable memory cardby way of example, the recording medium 90 may be any other recordingmedium for recording image data of a still image or a moving image to besaved as a result of image capture. For example, a portable disk mediumsuch as an optical disk may be used, or a hard disk drive (HDD) may beincorporated and used for recording.

The external interface 8 sends and receives various data to and from anexternal device via a predetermined cable according to a signal standardsuch as the universal serial bus (USB) standard. The external interface8 may be an external interface complying with a standard other than theUSB standard, such as the Institute of Electrical and ElectronicsEngineers (IEEE) 1394 standard.

In place of a wired transmission interface, the external interface 8 maybe a wireless transmission interface such as an infrared transmissioninterface or a near field communication interface.

The image capture apparatus 1 is configured to send and receive data toand from a personal computer or other various devices via the externalinterface 8. For example, the image capture apparatus 1 can transfercaptured image data or image data obtained as a result of thecombination process to an external device.

In the image capture apparatus 1 of the present embodiment, the CPU 31executes image capture operation control or arithmetic processing andcontrol for various operations described below according to a programstored therein. FIG. 3 shows operation functions implemented by thearithmetic processing of the CPU 31. Specifically, an image capturecontrol unit 51, a pre-combination processing unit 52, a combinationprocessing unit 53, a recording/playback/transmission control unit 54,an operation detection unit 55, a display control unit 56, and atemplate management unit 57 are formed as software function blocks.

The image capture control unit 51 performs image capture operationcontrol. Specifically, the image capture control unit 51 controls eachunit in the image capture system 2 or the camera DSP 4 to capture asubject image. The image capture control unit 51 performs otherprocesses such as an autofocus process, an automatic exposure adjustmentprocess, and a flash light emission control process.

The pre-combination processing unit 52 may perform a pre-combinationprocess so that a plurality of frames of image data having continuity intime that are used for a combination process described below are used asimage data to be combined (hereinafter referred to as “combination-useimage data”). For example, the pre-combination processing unit 52 mayobtain a plurality of frames of image data having continuity in timethat are recorded on the recording medium 90, and use the obtainedframes of image data as combination-use image data. The term “aplurality of frames of image data having continuity in time” means aplurality of consecutive or intermittent frames of image data that canbe extracted from a plurality of frames obtained by a series oftemporally consecutive image capture actions. The pre-combinationprocessing unit 52 may obtain a sequence of image data supplied from anexternal device as a target for the combination process.

The pre-combination processing unit 52 may further perform a brightnessadjustment process on image data of each of the plurality of frames ofimage data having continuity in time used as combination-use image data.The brightness adjustment process may be a process of equalizing averagebrightness levels of image data of all or some of the frames of imagedata used as the combination-use image data. In particular, frames ofimage data captured without using a flash can be extracted from amongthe plurality of frames of image data used as the combination-use imagedata, and the average brightness levels of the extracted image data canbe equalized.

The combination processing unit 53 may perform a combination process onthe combination-use image data obtained in the pre-combination processby the pre-combination processing unit 52 according to the operationinput information so as to generate combined-image data representing astill image.

For example, the combination processing unit 53 may perform acombination process on combination-use image data of frames in a rangeon a time axis specified by the operation input information among thecombination-use image data having continuity in time so as to generatecombined-image data representing a still image.

Further, the combination processing unit 53 may perform a combinationprocess on combination-use image data of each of the plurality of framesusing a weighting coefficient specified by the operation inputinformation so as to be generate combined-image data representing astill image.

The combination processing unit 53 may further perform a combinationprocess on combination-use image data of a plurality of frames usingweighted averages so as to generate combined-image data representing astill image.

In addition to the processes described above, the combination processingunit 53 may perform arithmetic operations for various combinationprocesses described below.

The recording/playback/transmission control unit 54 instructs the mediainterface 10 to control reading from the recording medium 90 or writinginto the recording medium 90. For example, therecording/playback/transmission control unit 54 may instruct the mediainterface 10 that a plurality of frames of image data having continuityin time that are recorded on the recording medium 90 be read so as to beobtained by the pre-combination processing unit 52.

The recording/playback/transmission control unit 54 may further performa process of recording the combined-image data generated by thecombination processing unit 53 onto the recording medium 90 or sendingthe combined-image data to an external device via the external interface8.

The operation detection unit 55 detects operation input informationprovided by the user. Specifically, the operation detection unit 55detects input information from the operation unit 5. Based on theoperation input information detected by the operation detection unit 55,the image capture operation control in the image capture control unit51, the pre-combination process in the pre-combination processing unit52, the combination process in the combination processing unit 53, andthe control process in the recording/playback/transmission control unit54 are executed.

The display control unit 56 instructs the display controller 7 toexecute necessary display on the display panel 6. For example, monitordisplay during image capture or display of a playback image is executed.

In the combination process of the combination processing unit 53, thedisplay control unit 56 generates and outputs combination-work imagedata to the display controller 7 to display a combination-work image onthe display panel 6.

The combination-work image data may be display data including a playbackmoving image including the combination-use image data of the pluralityof frames obtained in the pre-combination process, an image used tospecify a range on a time axis, the range being a range ofcombination-use image data used in a combination process among thecombination-use image data of the plurality of frames, an imageincluding representations of weighting coefficients each assigned tocombination-use image data of one of a plurality of frames, and acombined image generated in a combination process performed usingcombination-use image data of a plurality of frames (such as a previewimage obtained as a result of the combination process). When a givencombination process is performed, the display control unit 56 canfurther generate the combination-work image data so as to include bothimages obtained before and after the given combination process, and canoutput the generated combination-work image data to the displaycontroller 7 to display the corresponding image on the display panel 6.

The display control unit 56 can further output combined-image datafinally generated in the combination process to the display controller 7as image data for display, and can display the corresponding image onthe display panel 6.

The template management unit 57 is configured to manage coefficienttemplates that are prepared to simplify user operations in thecombination process and to select a coefficient template.

As described below, a weighting coefficient can be assigned to each of aplurality of frames of image data to be combined. For example, weightingcoefficient patterns for achieving image effects such as thefirst-curtain synchronization effect, second-curtain synchronizationeffect, and multi-flash effect are created as templates which are storedin, for example, the flash ROM 33 or the like. In a case such as whenthe operation detection unit 55 detects a user operation or when animage for selection from the templates is displayed by using the displaycontrol unit 56, the template management unit 57 performs a process ofselecting a coefficient template according to the user operation andsending the selected coefficient template to the combination processingunit 53 so that the selected coefficient template can be used in thecombination process.

2. Operation Mode

FIG. 4 shows operation modes of the image capture apparatus 1 of thepresent embodiment. In the image capture apparatus 1 of the presentembodiment, the operation mode is changed to a camera mode, a playbackmode, and a combination mode in accordance with a user operation. Inpractice, other modes such as a communication mode that allows forcommunication with an external device may be provided, which are omittedfor simplicity of illustration.

The camera mode is a mode in which the image capture system 2 performsimage capture. That is, the camera mode is an operation mode in which auser normally captures images using a camera. In the camera mode, avariety of image capture modes as shown in FIG. 4 are available.

Combination-mode image capture is an image capture mode in which animage used for a combination process described below is captured, whichis a characteristic operation of the present embodiment. A plurality offrames having continuity in time are captured, and the plurality offrames of image data are recorded onto the recording medium 90.

In the following description, image data obtained as a result of imagecapture is stored onto the recording medium 90 in the image captureoperation. The image data may be stored in the flash ROM 33 instead ofthe recording medium 90. Another operation method in which the imagedata is normally recorded onto the recording medium 90 and is recordedonto the flash ROM 33 when the recording medium 90 is not placed may beused.

An auto mode is an image capture mode in which the optimum settings suchas the aperture value, shutter speed, and International Organization forStandardization (ISO) sensitivity are automatically performed using theimage capture apparatus 1.

A portrait mode is an image capture mode in which image capture isperformed with settings optimal for providing shots of people.

A landscape mode is an image capture mode in which image capture isperformed with settings optimal for providing shots of landscapes.

A macro mode is an image capture mode in which image capture isperformed with settings optimal for providing shots of subjects closerto the camera than normal. A special macro mode, namely, a nature macromode for providing sharp and vivid macro shots of flowers and insects,may be provided.

A sports mode is an image capture mode in which image capture isperformed with settings optimal for providing shots of actions.

A sunset scene mode is an image capture mode in which image capture isperformed with settings optimal for providing shots of sunset scenes.

A night scene mode is an image capture mode in which image capture isperformed with settings optimal for providing shots of night scenes.

A movie mode is an image capture mode in which a moving image iscaptured.

Other modes, such as a night portrait mode suitable for providing shotsof people in night scenes and a firework mode suitable for providingshots of fireworks, may be provided.

After setting the camera mode, the user selects a desired image capturemode from among the image capture modes described above, therebyperforming an image capture operation to obtain a captured imagesuitable for the subject type or condition.

In order to obtain an image with a special effect such as the long-timeexposure effect, first-curtain synchronization effect, second-curtainsynchronization effect, or multi-flash effect described below, the userexecutes the combination-mode image capture to create an image with adesired effect in a subsequent combination process.

In the camera mode, the image capture control unit 51, the operationdetection unit 55, the display control unit 56, and therecording/playback/transmission control unit 54 in the CPU 31 cooperatewith each other to control the image capture system 2, and control therecording operation and the display operation to capture an image,display an image of captured image data, and record the captured imagedata.

The playback mode is an operation mode in which an image captured andrecorded on the recording medium 90 or the flash ROM 33 is played back.

In accordance with a user operation, the CPU 31 performs control to readan image recorded on the recording medium 90 or the flash ROM 33 and toplay back and display the image on the display panel 6.

In the playback mode, the operation detection unit 55, the displaycontrol unit 56, and the recording/playback/transmission control unit 54in the CPU 31 cooperate with each other to control the playbackoperation and the display operation to execute the display or the likeof the image data.

The combination mode is an operation mode in which a plurality of framesof image data having continuity in time are used as combination-useimage data to perform a combination process. The CPU 31 advances thecombination process in accordance with a user operation.

In the combination mode; in steps ST1 and ST2, the CPU 31 performs apre-combination process. First, in step ST1, the CPU 31 performs atarget image selection/acquisition process. For example, a sequence ofimage data having continuity in time captured in the combination-modeimage capture and recorded on the recording medium 90 is selected andacquired as combination-use image data used in a combination process.

In step ST2, the CPU 31 performs a combining preparatory process. Asdescribed below, the combining preparatory process is a process forperforming, for example, brightness adjustment (exposure adjustment) ofthe acquired sequence of image data and equalizing the brightness statesof the image data elements of the sequence of image data so that theimage data elements become in a state suitable for the combinationprocess.

After the pre-combination process in steps ST1 and ST2 has beencompleted, in step ST3, the CPU 31 performs a combination process. Inthis case, the CPU 31 advances the combination process using a sequenceof combination-use image data in accordance with a user operation. Thiscombination process will also be described in detail below.

When combined-image data as a final result of combination is generatedusing the combination process, in step ST4, the CPU 31 performs acombined-image recording process.

Specifically, the CPU 31 records the combined-image data generated usingthe combination process onto the recording medium 90 through the mediainterface 10. Alternatively, the combined-image data may be recordedonto the flash ROM 33. Thus, thereafter, the user can play back thecombined image as desired.

The CPU 31 may also send the combined-image data to an external deviceconnected to the external interface 8. Thus, for example, thecombined-image data can be recorded onto a predetermined recordingmedium or the corresponding image can be displayed and output using theexternal device.

In the combination mode, in steps ST1 and ST2, the pre-combinationprocessing unit 52, the operation detection unit 55, the display controlunit 56, and the recording/playback/transmission control unit 54 in theCPU 31 cooperate with each other to perform a necessary processingoperation.

In step ST3, the combination processing unit 53, the operation detectionunit 55, and the display control unit 56 in the CPU 31 cooperate witheach other to perform a necessary processing operation. In a case wherecoefficient templates are used, the template management unit 57 works.

In step ST4, the combination processing unit 53, the operation detectionunit 55, the display control unit 56, and therecording/playback/transmission control unit 54 in the CPU 31 cooperatewith each other to perform a necessary processing operation.

3. Camera-Mode Processes

First, the processes of the CPU 31 in the camera mode will be describedwith reference to FIG. 5.

In steps F10, F11, and F12, the CPU 31 monitors a user operation.

In step F10, the CPU 31 monitors the operation of the image capturemode. As described above, a variety of modes such as the combinationmode, the auto mode, and the portrait mode are available as imagecapture modes, and the user can select a desired image capture modeaccording to the image capture purpose. A mode may be selected by, forexample, operating the dial operation unit 5 h shown in FIG. 2B orselecting a mode in the menu display on the display panel 6.

When the user selects an image capture mode, the CPU 31 proceeds to stepF13 and performs operation settings according to the selected imagecapture mode. For example, the CPU 31 controls each unit in the imagecapture system 2 to determine settings such as the exposure amount, theexposure method, the electronic shutter setting, the frame rate, and thegain. In particular, when the combination-mode image capture isselected, the setting of continuous image capture is also performed.

In step F11, the CPU 31 monitors a shutter operation performed by theuser. When a shutter operation performed by the user is detected, instep F14, the CPU 31 causes the process to branch depending on whetheror not the current image capture mode is the combination-mode imagecapture.

If the current image capture mode is other than the combination-modeimage capture, in step F16, an image capture process is performed.Specifically, the CPU 31 controls the camera DSP 4, the media interface10, and any other suitable unit so that an image of one frame obtainedat the time of the shutter operation is stored as captured image data,namely, still-image data.

The combination-mode image capture is an image capture mode forobtaining image data used for a later combination process. When ashutter operation is performed in the combination-mode image capture,the CPU 31 proceeds to step F15 and executes a continuous image captureprocess as combination-mode image capture. The continuous image captureprocess may be a movie-like capture operation in which each of framesconsecutively obtained by the image capture system 2 is stored as imagedata onto the recording medium 90 or the like. Depending on the settingof the frame rate, for example, intermittent frames such as every otherframe may be stored. The captured image data is stored as a large numberof still images or a moving image.

While, in step F15, frames are consecutively stored as captured imagedata, image capture may be executed for a period of time as follows byway of example:

(1) Frames of image data for a period from when the user performs ashutter operation to when the user performs a second shutter operationare stored as captured images.

(2) Frames of image data for a period from when the user performs ashutter operation to when the time counted by the CPU 31 has reached apreset timer value are stored as captured images. The preset timer valuemay be fixed or selected by the user.

(3) Frames of image data for a period during which the user continues ashutter operation (continues to press the shutter release button) arestored as captured images. When the user releases the shutter releasebutton, the image capture ends.

By performing the combination-mode image capture in step F15, aplurality of frames of image data having continuity in time are capturedand stored.

The plurality of frames of image data are stored in association witheach other so as to be identified as a sequence of image data havingcontinuity in time.

For example, the image data of each frame may be assigned a serialnumber and may be recorded, and management information regarding a rangeincluding the serial numbers may be added. Alternatively, the image dataof the frames may be associated with each other by adding metadataindicating image data obtained by the series of image capture actions.

Such association may not necessarily be performed during image capture.In a combination process described below, image data of framesconsecutive in order image-capture time may be extracted to read theimage data of the consecutive frames. Time and date information obtainedin the time counting by the clock circuit 34 is added to image data ofeach frame to be captured and recorded. Thus, such association in thereading operation can also be performed.

Further, metadata is added to image data of each frame to be captured.The metadata may be information regarding emission of flash light fromthe flash light emitting unit 15. The flash light emission is controlledby the CPU 31 according to a user operation, a setting, or the like.

Information obtained in the image capture operation may also be added asmetadata to image data of each frame, such as information regarding thedistance to the main subject, which is measured by the lens positiondetection unit 27, and the distance distribution information generatedby the information generation unit 44.

The user can perform various operations other than theimage-capture-mode operations and the shutter operation. If any of suchother operations is detected in step F12, the CPU 31 proceeds to stepF17 and executes a process corresponding to the detected operation. Forexample, when the operation of changing the mode to the playback mode orthe combination mode is performed, a mode change process is performed.The CPU 31 performs necessary processes according to various operations,such as various settings for image capture, for example, the manualsetting of the exposure amount, the zoom operation, the manual focussetting, and the setting of use of flash/non-use of flash/automaticlighting.

In the camera mode, image capture is performed in various image capturemodes according to the process shown in FIG. 5 described above, andcaptured image data is stored on the recording medium 90.

FIG. 6 shows an example in which various types of image data stored onthe recording medium 90 are played back and displayed as images.Together with image data of images PCT1, PCT2, and PCT4 captured in thenormal image capture modes (such as the auto mode and the portraitmode), an image data set captured in the combination-mode image captureselected by the user, such as image data of an image PCT3, is stored onthe recording medium 90. The user performs an operation in the playbackmode to play back and display the corresponding images on the displaypanel 6 for confirmation.

The image data of the image PCT3 captured in the combination-mode imagecapture is a group of image data elements associated as havingcontinuity in time, as indicated by, for example, image data elements#0, #1, and #2.

When the user plays back and views the captured images in the playbackmode, the image data of the images PCT1, PCT2, PCT3, PCT4, etc., aresequentially played back. For the image data of the image PCT3 obtainedin the combination-mode image capture, a representative image among theactually recorded image data elements #0, #1, #2, etc., may be playedback and displayed. For example, the top image data element #0 may bedisplayed.

Furthermore, in order to help the user understand that a large number ofimage data elements (#0, #1, #2, etc.) are actually stored in thecombination-mode image capture, as shown in FIG. 6, a mark MK indicatingimage data captured in the combination-mode image capture is displayedon a display screen during playback.

The user can use the mark MK to understand that the image data of theimage PCT3 actually includes a large number of temporally consecutiveimages which can be used for a combination process.

That is, after performing combination-mode image capture, the user playsback images and selects an image with the mark MK to perform acombination work as an operation in a combination mode described below.

4. Combination-Mode Process

4-1: Combining Preparatory Process

A process performed in the combination mode will now be described.

As described above with reference to FIG. 4, in the combination mode,the CPU 31 performs the target image selection/acquisition process(ST1), the combining preparatory process (ST2), the combination process(ST3), and the combined-image recording process (ST4).

The target image selection/acquisition process (ST1) and the combiningpreparatory process (ST2) are pre-combination processes prior to anactual combination process. First, the CPU 31 (pre-combinationprocessing unit 52) captures combination-use image data in the targetimage selection/acquisition process (ST1).

For example, as shown in FIG. 6, when a user performs the operation ofplaying back captured images, selecting an image captured in thecombination-mode image capture (for example, the image PCT3 shown inFIG. 6), and instructing a combination process, the CPU 31 starts theprocess in the combination mode. In this case, the CPU 31 captures theimage selected by the user operation as combination-use image data.Specifically, when the image PCT3 shown in FIG. 6 is selected, the CPU31 reads the sequence of image data elements #0, #1, #2, etc., as havingcontinuity in time, which are associated with the image PCT3, from therecording medium 90, and acquires them as combination-use image data.Then, the image data elements #0, #1, #2, etc., are set as image data tobe combined.

In the combining preparatory process (ST2), a brightness adjustmentprocess is performed on the captured combination-use image data (imagedata elements #0, #1, #2, etc). In this process, the exposure amounts(screen intensities) of the image data elements #0, #1, #2, etc., areequalized.

In this case, for example, the CPU 31 (pre-combination processing unit52) performs a process shown in FIG. 7.

First, in step F101, the CPU 31 extracts a non-flash image from theimage data elements #0, #1, #2, . . . , #n captured as targets forcombination. The non-flash image is image data that has been capturedwithout using a flash. As described above, since information regardingemission of flash light during the image capture is added as metadata toimage data, the CPU 31 can extract a non-flash image by checkingmetadata of the captured image data elements #0, #1, #2, . . . , #n.

Note that the non-flash image is based on information as to whether ornot the device that has performed the image capture (i.e., the imagecapture apparatus 1) has performed the image capture using a flash(hereinafter referred to as “self-flashing image capture”), and does notinclude an image caused by the screen intensity that suddenly becomes ashigh as that in the flash light emission state.

For example, an image of a subject which is illuminated with flash lightby another photographer or illuminated by the headlight of a vehiclepassing by during the image capture is not an image obtained usingself-flashing image capture and is handled as a non-flash image.

If, as shown in FIG. 7, among image data elements #0, #1, . . . , #n,self-flashing image capture is performed at the time of the image dataelements #2 (that is, if a flash light emission indication is given asmetadata of the image data element #2), the image data elements #0, #1,#3, . . . , #n, except for the image data element #2, are extracted.

Then, in step F102, the CPU 31 calculates average brightness values forthe extracted image data elements (for example, the image data elements#0, #1, #3, . . . , #n).

In this case, an average value of brightness signal values of all pixelsmay be calculated for each image data element, or weighting may beperformed on each area within the screen and a weighted averagebrightness value may be calculated. In this case, an average brightnessvalue for each image data element may be calculated using a techniquesimilar to that for calculating a brightness level for the so-calledautomatic exposure adjustment.

In step F103, the CPU 31 classifies the image data elements by averagebrightness value, and selects a representative group. For example, asshown in FIG. 7, groups Yg0, Yg1, . . . , Yg5 are set in steps ofbrightness levels, starting from the low-brightness level side to thehigh-brightness level side, and the image data elements are classifiedinto the groups depending on the average brightness value. For example,in FIG. 7, it is assumed that 28 image data elements are present asextracted image data elements #0, #1, #3, . . . , #n and that 25 imagedata elements are classified as the group Yg1, two image data elementsas the group Yg4, and one image data element as the group Yg5. Arepresentative group is selected based on the result of theclassification. The group Yg1 is used as a representative group, by wayof example.

A sequence of combination-use image data is a plurality of frames ofimage data consecutive in time, and is continuously captured duringimage capture in step F15 shown in FIG. 5. Thus, no extreme differencein brightness level generally occurs. Hence, frames of image dataobtained without using self-flashing image capture are generallyclassified as the same group. However, in some cases, the subjectbrightness level may suddenly largely change during continuous imagecapture because of accidental reasons such as the firing of a flash byanother photographer, instantaneous illumination of the subject usingthe headlight of a vehicle passing by, and beams of light that suddenlybreak through the clouds. As a result of the classification shown inFIG. 7, the images classified as the groups Yg4 and Yg5 are consideredas images obtained due to such instantaneous change in exposure amountdue to the reasons described above.

If such images having a large difference in subject brightness level areincluded, only the images may be highlighted or the like in thecombination process, which may fail to implement combination as desired.

Thus, the brightness adjustment is performed for each image data elementin step F104 and the subsequent steps.

First, in step F104, the CPU 31 calculates an average value of averagebrightness values of image data elements included in the representativegroup (Yg1). The calculated average value is set as a reference averagebrightness level.

In step F105, the CPU 31 selects one (for example, the image dataelement #0) of the image data elements #0, #1, #3, . . . , #ncorresponding to the non-flash images extracted in step F101 as a targetfor correction. In step F106, the CPU 31 determines the ratio of theaverage brightness value of the image data element #0 selected as thetarget to the reference average brightness level, and calculates acorrection coefficient according to the ratio.

Then, in step F107, the CPU 31 multiplies the brightness values of allpixels of the image data element #0 by the correction coefficient toperform brightness correction of the image data element #0.

In step F108, if an unprocessed image data element remains, the CPU 31returns to step F105 and performs similar processing on a next imagedata element (for example, the image data element #1).

Specifically, in steps F105, F106, and F107, the brightness correctionis sequentially performed on all the image data elements extracted asnon-flash images. When the process is completed for all the image dataelements, the process shown in FIG. 7 ends.

According to the combining preparatory process shown in FIG. 7, thebrightness levels of all image data elements extracted as non-flashimages acquired as combination-use image data are equalized to produceimage data suitable for a combination process described below.

In this embodiment, no correction is performed on an image obtainedusing self-flashing image capture because the self-flashing imagecapture has been intentionally performed by the photographer of theimage.

In some cases, all combination-use image data may be corrected toequalize the brightness levels regardless of flashlight emission.

In other cases, if a plurality of frames of image data captured using aflash are included, the plurality of frames of flash image data may becorrected to equalize the brightness levels.

4-2: Combination Process

The combination process in step ST3 shown in FIG. 4 will now bedescribed. In the combination process of the present embodiment, imageprocessing is performed so that, instead of simply adding the values ofcorresponding pixels of a plurality of frames of image data, the valuesof the corresponding pixels are averaged (or weighted) before a combinedimage is generated.

In the combination process in step ST3, the combination processing unit53, the display control unit 56, and the operation detection unit 55 inthe CPU 31 cooperate with each other to perform a process shown in FIG.8.

First, in step F201, the CPU 31 starts the playback and display of thecombination-use image data using a combination-work image.

FIGS. 10 to 18 show examples of a combination-work image 70 displayed onthe display panel 6.

At the start of the playback, initially, the combination-work image 70shown in FIG. 10 is displayed. The combination-work image 70 shown inFIG. 10 includes image display areas 71 and 72, a timeline 73, and aplayback position marker 74.

The image display area 71 is used for the display or the like of aplayback image.

The image display area 72 is used for the preview display or the like ofa combined image.

The timeline 73 represents a time width of combination-use image data ofa plurality of temporally consecutive frames.

The playback position marker 74 represents a current playback positionon the timeline 73.

At the start of the playback, initially, as shown in FIG. 10, images inthe combination-use image data are sequentially played back anddisplayed in the image display area 71 in a movie fashion, starting fromthe temporally first image. For example, if it is assumed that thecombination-use image data includes image data elements #0, #1, #2, . .. , #n, the CPU 31 sequentially displays images corresponding to theimage data elements #1, #2, #3, . . . in the image display area 71 insequence, starting from the image data element #0, so that the sequenceof image data elements #0 to #n can be represented in a movie fashion.The playback position marker 74 moves to the left along the timeline 73with the progress of the playback.

A user specifies a combination start position at a certain time whileviewing the playback image. For example, a user may specify acombination start position by pressing the shutter operation key.

After the playback and display are started in step F201, in step F202,the CPU 31 sets a combination start position in accordance with anoperation performed by the user for specifying a combination startposition.

For example, as shown in FIG. 11, at a time point in the course of theplayback, the user specifies a combination start position. Then, animage data element #x corresponding to a playback image at the timepoint is set as an image at the combination start position (hereinafterreferred to as “combination-start image”). A combination start marker 75is further displayed.

The playback and display of images in the image display area 71 arecontinuously performed.

After the combination start position is set, the CPU 31 continues theprocessing of steps F203 and F204 until the operation of setting acombination end position is detected in step F205.

The preview image combination in step F203 is a process of combining thecombination-start image up to the image currently being played back inthe image display area 71.

The preview image display in step F204 is a process of displaying acombined image in the image display area 72 as a preview.

Specifically, all images from the combination-start image to the framecurrently being played back are combined, and a resulting combined imageis displayed in the image display area 72 as a preview in the mannershown in FIG. 12.

Since the playback is still in progress, the size of a combination rangeincreases by one frame with the progress of the playback. Specifically,each time a playback image is played back, the values of thecorresponding pixels are added, and the resulting sum is divided by thenumber of playback images played back. Such combination is performed togenerate a preview image. Thus, a non-weighted image similar to an imagecaptured using long-time exposure is obtained and is displayed as apreview image.

In a case where the sequence of image data elements #0 to #n representsan image in which a subject is moving from right to left, as shown inFIG. 12, in accordance with the progress of the playback in the imagedisplay area 71, a long-time exposure image in which the subject ismoving from right to left is displayed in the image display area 72 as acombined image obtained by combining the combination-start image up tothe current image.

The user specifies-a combination end position while viewing the playbackand preview images.

The user may set a combination end position by, for example, pressingthe shutter operation key to specify a combination start position andthen releasing the shutter operation key.

After the shutter operation key is pressed, the CPU 31 repeats theprocessing of steps F203 and F204. When the shutter operation key isreleased, the playback image obtained at this time is set as an image atthe combination end position (hereinafter referred as a “combination-endimage). Then, as shown in FIG. 12, a combination end marker 76indicating the combination end position is displayed. Thereafter, theCPU 31 proceeds from step F205 to step F206.

The operation of setting the combination start position/combination endposition may be performed by, but not limited to, pressing the shutteroperation key. For example, the shutter operation key may be pressedonce to set a combination start position and the shutter operation keymay be pressed again to set a combination end position.

Alternatively, combination-use image data of consecutive frames may beselected in advance from the first frame to the last frame, or acombination start position may be set by a user operation and acombination end position may be set when the capacity of a buffer memoryused for preview image or image combination becomes full.

The generation and display of a preview image may be performed with asimple setting (such as a weighting setting) before a final imagecombination process is performed in step F207, which will be describedbelow, or may be simply performed with an image size enough to allow auser to confirm the effect of the setting in advance.

When the combination start position and the combination end position areset in the processing up to step F205, the CPU 31 proceeds to step F206and performs an adjustment process. FIG. 9 shows the details of theadjustment process.

First, in step F220, the CPU 31 displays a selection image list andweight bars in the combination-work image 70.

As shown in FIG. 13, images in the combination range from thecombination-start image to the combination-end image are displayed in aselection image list view in the timeline 73.

For example, in this example, the image data element #5 is used as acombination-start image and the image data element #11 is used as acombination-end image, by way of example. The image data elements #5,#6, #7, . . . , #11 are displayed in the selection image list view inthe timeline 73.

In practice, although all images in the combination range may notdisplayed in the timeline 73, the selection image list is automaticallyresized so as to be displayed in the timeline 73. In terms ofvisibility, for example, the selection image list may be scrollableright and left to show the images to avoid excessive reduction in size.Alternatively, preceding and following frames of an image to be selectedmay also be displayed side-by-side in the selection image list so thatthe user can easily check a frame to be selected in the adjustmentoperation.

When weight bars are displayed, weight bars w5, w6, w7, . . . , w11corresponding to the image data elements #5, #6, #7, . . . , #11,respectively, are displayed. For example, each weight bar has a heightthat represents a weighting coefficient.

In the initial state in step F220, no weight is assigned to a specificimage data element. In other words, respective image data elements areequally weighted.

In the present embodiment, in the continuous description, a weight baris adapted to represent a weighting coefficient. An image representing aweighting coefficient assigned to each image data element is not limitedto the weight bar. For example, any other shape (such as a circulargraph shape) or a numerical value indicating a weighting coefficient maybe displayed.

Alternatively, an independent image representing only a weightingcoefficient may not necessarily be displayed. For example, an imagerepresenting a weighting coefficient assigned to each image data elementmay be implemented using a technique such as changing the lightnesslevels of the images corresponding to the image data elements #5, #6,#7, . . . , #11 displayed in the selection image list according to theweighting coefficients.

In the process shown in FIG. 9, the CPU 31 monitors a user operation insteps F221, F222, and F223.

In the combination-work image 70 shown in FIG. 13, a user can performthe operation of changing the weighting coefficients of the image dataelements, the operation of changing the combination range, or any othersuitable operation.

For example, the user can perform the operation of performing aleft-right operation of the cross key 5 i shown in FIG. 2B to selectimage data of a desired image in the selection image list and thenperform an up-down operation of the cross key 5 i to change theweighting coefficient of the selected image data.

When the user performs the operation of selecting a certain image andchanging the corresponding weighting coefficient, the CPU 31 advancesthe process from step F221 to step F224.

In the display process in accordance with the operation, the CPU 31changes the height of the weight bar w(x) corresponding to the selectedimage data element #(x).

The CPU 31 further changes the weighting coefficient set for theselected image data element #(x).

The CPU 31 further performs, in addition to reflecting the changedweighting coefficient, a combination process using weighted averages ofthe image data elements within the combination range to generate apreview image, and displays the preview image in the image display area72.

For example, the user performs the operation of selecting the image dataelement #5 in the state shown in FIG. 13 and increasing thecorresponding weighting coefficient. Then, the CPU 31 performs theprocessing of step F224 to increase the weight bar w5 in the mannershown in FIG. 14, change the weighting coefficient set for the imagedata element #5 to a high value, and perform the combination process ofthe image data elements (#5 to #11) within the combination range toproduce a preview image which is then displayed in the image displayarea 72.

The user further performs the operations of selecting the remainingimage data elements #6 to #11 and reducing the corresponding weightingcoefficients. Then, the CPU 31 performs the processing of step F224 inaccordance with the individual operations to reduce the heights of theweight bars w6 to w11 in the manner shown in FIG. 14 to change theweighting coefficients set for the image data elements #6 to #11 to lowvalues and thereafter perform the combination process of the image dataelements (#5 to #11) within the combination range to produce a previewimage which is then displayed in the image display area 72.

When the user performs the weighting coefficient operations describedabove for the image data elements #5 to #11, combination is performed sothat the temporally first image data element #5 in the combination rangeis highlighted by a higher weight than the remaining image data elements#6 to #11. As indicated by the preview image shown in FIG. 14, acombined image with the so-called first-curtain synchronization effectis obtained.

The first-curtain synchronization effect is an image capture techniquein which a clear representation of the temporally first state isprovided by firing a flash only at the beginning of long-time exposure.In the present embodiment, the first-curtain synchronization effect isobtained by combining image data of consecutive frames in a combinationrange so that a high weighting coefficient is assigned to the image dataof the first frame and a low weighting coefficient is assigned to theimage data of the subsequent remaining frames.

The weighting coefficients can be changed with, for example, variationsof 8 bits (256 levels) to realize fine adjustment. For more simplicityof operations for users, two levels, namely, bright and dark, or threelevels, namely, upper, middle, and lower, may be used. The coefficientsetting of a weight of zero may be added in levels.

The weighting coefficients may be set so that an exposure amount, or thelightness of an image, can be adjusted as desired without removing anundesired image during continuous image capture (such as an imageobtained when the surroundings of a photographer who is capturing anight scene are illuminated by the headlight of a vehicle passing by).In the present embodiment, such an image as unsuitable for combinationis corrected in advance in the combining preparatory process describedabove. For example, the adjustment of weighting coefficients can beperformed manually by the user.

In a user operation, the combination range defined by the combinationstart marker 75 and the combination end marker 76 can be changed.

For example, when the image data element (#5) at the combination startposition is selected, the user can perform a left-right operation of thecross key 5 i, while pressing a specific key, to shift the combinationstart position to the temporally preceding or following frame of imagedata. Likewise, when the image data element (#11) at the combination endposition is selected, the user can perform a left-right operation of thecross key 5 i, while pressing a specific key, to shift the combinationend position to the temporally preceding or following frames of imagedata.

Alternatively, a user operation may be performed to directly move thecombination start marker 75 or the combination end marker 76 to the leftand right along the timeline 73.

When the user performs the operation of changing the combination range,the CPU 31 advances the process from step F222 to step F225. Then, inaccordance with the operation, the CPU 31 changes the image range of atarget for combination, performs a combination process of image dataelements in the new combination range to generate a preview image, anddisplays the preview image in the image display area 72.

For example, it is assumed that the user performs the operation ofchanging the combination-end image to the image data element #7 in thestate shown in FIG. 14. Then, the CPU 31 performs the processing of stepF225 to perform a combination process of the image data elements (#5 to#7) in the new combination range in the manner shown in FIG. 15 anddisplay a preview image in the image display area 72. The combinationend marker 76 is shifted so as to designate the image data element #7.

In this case, as can be seen from the comparison between the previewimages shown in FIGS. 14 and 15, in FIG. 15, due to the shortenedcombination range, an image effect with the exposure time reduced infirst-curtain synchronization is achieved.

As described above, the user performs various operations on thecombination-work image 70, thereby displaying various combined images aspreviews accordingly through the processing of steps F224 and F225performed by the CPU 31.

FIGS. 16, 17, and 18 show other examples.

FIG. 16 shows an example of the combination-work image 70 obtained when,for example, the user performs the operation of selecting the image dataelement #11 in the state shown in FIG. 13 and increasing thecorresponding weighting coefficient and also performs the operations ofindividually selecting the remaining image data elements #5 to #10 andreducing the corresponding weighting coefficients.

The CPU 31 performs the processing of step F224 in accordance with theindividual weighting coefficient operations for the image data elements#5 to #11.

In this case, combination is performed so that the temporally last imagedata element #11 in the combination range is highlighted by a higherweight than the remaining image data elements #5 to #10. As indicated bythe preview image shown in FIG. 16, a combined image with the so-calledsecond-curtain synchronization effect is obtained.

The second-curtain synchronization effect is an image capture techniquein which a clear representation of the temporally last state is providedby firing a flash only at the end of long-time exposure. In the presentembodiment, the second-curtain synchronization effect is obtained bycombining image data of consecutive frames in a combination range sothat a high weighting coefficient is assigned to the image data of thelast frame and a low weighting coefficient is assigned to the image dataof the remaining frames.

FIG. 17 shows an example of the combination-work image 70 obtained whenthe user performs the operation of changing the combination range in thestate shown in FIG. 16. When the user performs the operation of changingthe combination range to a range of the image data elements #9 to #11,the CPU 31 performs the processing of step F225 to perform a combinationprocess of the image data elements in the new combination range togenerate a preview image which is then displayed in the image displayarea 72.

In this case, as can be seen from the comparison between the previewimages shown in FIGS. 16 and 17, in FIG. 17, due to the shortenedcombination range, an image effect with the exposure time reduced insecond-curtain synchronization is achieved.

FIG. 18 shows an example of the combination-work image 70 obtained when,for example, the user performs the operation of increasing the weightingcoefficients of the image data elements #5, #8, and #11 in the stateshown in FIG. 13 and reducing the weighting coefficients of theremaining image data elements #6, #7, #9, and #10.

The CPU 31 performs the processing of step F224 in accordance with theindividual weighting coefficient operations for the image data elements#5 to #11.

In this case, combination is performed so that the images in thecombination range are periodically highlighted. As indicated by apreview image shown in FIG. 18, a combined image with the so-calledmulti-flash effect is obtained.

The multi-flash effect is an image capture technique in which a clearrepresentation of the state of a subject is provided by periodicallyfiring a flash during long-time exposure. In the present embodiment, themulti-flash effect is obtained by combining image data of consecutiveframes in a combination range so that high and low weightingcoefficients are periodically assigned to the image data.

As in the exemplary illustrations described above, a user can perform adesired operation on the combination-work image 70 to attempt thegeneration of various combined images and can confirm them as previewimages. That is, the user can easily generate a desired image byattempting visual observation of multiple image effects.

When a satisfactory combined image as a preview image is obtained, theuser may perform an adjustment termination process. The user may performthe adjustment termination operation by, for example, pressing a set keyat the center of the cross key 5 i.

When the adjustment termination operation is detected, the CPU 31 endsthe process shown in FIG. 9 from step F223, and proceeds to step F207shown in FIG. 8.

In step F207, the CPU 31 performs a final combination process.Specifically, the CPU 31 performs combination using the image dataelements in the combination range obtained at the end of the adjustmentprocess and the weighting coefficients set for the image data elements.In this case, the values of pixels of the image data of each of theframes in the combination range are multiplied by the set weightingcoefficients. Then, the values of pixels of the image data of each ofthe frames are added and the resulting sum is divided by the number offrames. That is, a weighted average process is performed.

Note that exposure correction may be applied in the addition or divisionprocess so that a combined image to be generated has specifiedlightness. Alternatively, the combined-image data obtained after thecalculation may be corrected.

After the combination process in step ST3 shown in FIG. 4 is performedusing the process shown in FIG. 8 described above, in step ST4, the CPU31 performs a combined-image recording process. Specifically, the CPU 31performs control so that the combined-image data generated in the mannerdescribed above is recorded onto the recording medium 90 through themedia interface 10.

The series of processes in the combination mode thus ends.

The operations in the combination mode described above can achieve thefollowing advantages:

First, effects equivalent to those obtained using long-time exposureimage capture, which is difficult in the related art because, due to theimage combination performed using images after they are captured, it isnecessary to determine the exposure time/shutter timing or the likeusing the photographer's experience and intuition, can easily beachieved even by inexperienced photographers. In addition, many attemptsor retries can be made until a satisfactory image effect is obtained.

Further, since weighting coefficients of image data elements to becombined after image capture can be set as desired, image effects suchas the first-curtain synchronization, second-curtain synchronization,and multi-flash effect can be easily achieved, which is difficult toachieve without using a flash in the related art.

Furthermore, a favorable image can be achieved even under imageconditions with motion of a subject, such as an image capture conditionthat aims to provide a representation of the motion of a subject or tofocus on a stationary subject among moving subjects.

4-3: Exemplary Adjustment Process Using Displayed Images Before andAfter Change

In the exemplary combination process described above, a preview image ofa combined image obtained when a user changes weighting coefficients orthe like in the combination-work image 70 is displayed in the imagedisplay area 72. Preferably, a combined image obtained immediatelybefore the weighting coefficients or the like have been changed(previous preview image) is also displayed at the same time to allow theuser to simultaneously view the combined images obtained before andafter the changing operation.

Accordingly, the adjustment process performed in step F206 shown in FIG.8 may be performed in a manner shown in FIG. 19 instead of using theexample shown in FIG. 9.

FIG. 19 shows an exemplary adjustment process executed by the CPU 31. InFIG. 19, the same or similar processing steps as or to those of FIG. 9are assigned the same numerals and descriptions thereof are omitted.

FIG. 19 is different from FIG. 9 in the processing of step F224A that isperformed when the user performs the operation of changing a weightingcoefficient and the processing of step F225A that is performed when theuser performs the operation of changing a combination range.

When the user performs the operation of changing a weightingcoefficient, in step F224A, the CPU 31 changes the height of the weightbar w(x) corresponding to the image data element #(x) being selected.The CPU 31 also changes the weighting coefficient set for the image dataelement #(x) being selected. The CPU 31 further performs, in addition toreflecting the changed weighting coefficient, a combination processusing weighted averages of the image data elements in the combinationrange to generate a preview image, and displays the preview image in theimage display area 72.

The CPU 31 further displays the previous combined image in the imagedisplay area 71, which has been displayed immediately before thisprocess in the image display area 72 as a preview image.

Specifically, the combined image displayed as the previous preview imageis not discarded but is stored, and is displayed, as shown in FIG. 20,together with the combined image obtained as the current preview imagein the image display areas 71 and 72 side-by-side, respectively.

FIG. 20 shows a combination-work image 70 obtained when, for example,the user performs the operation of increasing the weighting coefficientof the image data element #11. In this case, the current combined image(image with the second-curtain synchronization effect) is displayed inthe image display area 72 as a preview image obtained after the weightedchange, and the previous combined image (image with the second-curtainsynchronization effect removed) is displayed in the image display area71.

The combined images obtained before and after the change of weightingcoefficients are displayed at the same time in the manner shown in FIG.20, thus allowing the user to easily check the appropriateness of thechanged weighting coefficients.

Further, when the user performs the operation of changing a combinationrange, in step F225A, the CPU 31 changes the range of images to becombined in accordance with the operation, and performs a combinationprocess for image data elements in a new combination range to generate apreview image, and displays the preview image in the image display area72. The previous combined image is further displayed in the imagedisplay area 71, which has been displayed immediately before thisprocess in the image display area 72 as a preview image.

Also in this case, the user can compare the original and changedcombination ranges.

For example, in this manner, images obtained before and aftercombination are checked by comparison when a range of images to becombined is selected or a weight is specified. This is useful for theuser to attempt the implementation of various image effects.

For example, when the user compares two images and determines that theoriginal image is better, the operation of canceling the current imagecombination and recovering the original combination state may beperformed.

Furthermore, the user may be allowed to select, as desired, the processshown in FIG. 19 for displaying images obtained before and aftercombination side-by-side or the process shown in FIG. 9 described above.

5. Template-Based Process

In the combination process described above, a user can create a combinedimage with an image effect such as the first-curtain synchronizationeffect, second-curtain synchronization effect, or multi-flash effect byperforming the operation of changing a weighting coefficient of eachimage data element as desired. Some users may not be able to determinethe values of weights to obtain a desired image effect. Even users whoare able to determine the values of weights may feel that the operationof selecting image data elements one by one and changing the weightingcoefficients is time-consuming.

Accordingly, an operation technique in which coefficient templates eachhaving a pattern of weighting coefficients for achieving a predeterminedimage effect are prepared so that the user can select one of thecoefficient templates may be used.

In the following description, it is assumed that four coefficienttemplates for the long-time exposure effect, first-curtainsynchronization effect, second-curtain synchronization effect, andmulti-flash effect are prepared.

For example, weighting coefficient patterns corresponding to thelong-time exposure effect, first-curtain synchronization effect,second-curtain synchronization effect, and multi-flash effect are storedas coefficient templates in the flash ROM 33. For example, thecoefficient template for the first-curtain synchronization effectrepresents a weighting coefficient pattern in which a high weight isassigned to the first image while low weights are assigned to subsequentimages. The template management unit 57 of the CPU 31 manages suchcoefficient templates.

The CPU 31 performs the adjustment process in step F206 shown in FIG. 8in a manner shown in FIG. 22 instead of using the example shown in FIG.9.

In step F206 shown in FIG. 8, the CPU 31 performs the process shown inFIG. 22. First, in step F301, the CPU 31 (the template management unit57 and the display control unit 56) displays coefficient templates forselecting an effect on the display panel 6.

FIG. 21 shows an example of the display. Images corresponding to fourcoefficient templates for the long-time exposure effect, first-curtainsynchronization effect, second-curtain synchronization effect, andmulti-flash effect are displayed.

At this time, effect modeling images each indicating an image effectthat is obtained using the corresponding coefficient template aredisplayed. As shown in FIG. 21, an image with the long-time exposureeffect, an image with the first-curtain synchronization effect, an imagewith the second-curtain synchronization effect, and an image with themulti-flash effect are used as effect modeling images, thus allowing theuser to recognize the details of the image effects. This is suitablefor, in particular, users who are not familiar with various imageeffects.

The images for selection from the coefficient templates may be displayedas a list of preset effect modeling images each having image datarepresenting one of the coefficient templates.

Alternatively, text items “long-time exposure effect”, “first-curtainsynchronization effect”, “second-curtain synchronization effect”, and“multi-flash effect” may be presented in a menu without displayingeffect modeling images, and one of them may be selected.

In response to the display as shown in FIG. 21, for example, the useroperates the cross key 5 i to select a desired effect modeling image andperforms a determination operation to select one coefficient template.

In accordance with the selection and determination operation by theuser, the CPU 31 advances the process from step F302 to step F303. Inthis case, the template management unit 57 of the CPU 31 applies theweighting coefficient pattern of the selected coefficient template tothe combination processing unit 53, and the combination processing unit53 applies the weighting coefficient pattern to the individual images inthe combination range.

As described above, the CPU 31 proceeds to the adjustment process instep F206 shown in FIG. 8, which is shown in FIG. 22, when, as describedwith reference to up to FIG. 12, the user determines a combination range(combination start position and combination end position), that is, atthe time when the combination-work image 70 is changed from the stateshown in FIG. 12 to the state shown in FIG. 13.

When the user selects a coefficient template, as shown in FIG. 13, whilethe selection image list and the weight bars are displayed, thecombination processing unit 53 of the CPU 31 applies the weightingcoefficient pattern of the template as the weighting coefficientscorresponding to the image data elements, and displays, as a previewimage, a combined image obtained using weighted averages with theweighting coefficients applied.

The processing subsequent to the processing of steps F220 to F223 shownin FIG. 22 is similar to that shown in FIG. 9, and a redundantdescription thereof is omitted.

In this manner, the user selects a coefficient template, therebyinitially achieving a preview image of a desired image effect withoutadjusting weighting coefficients of image data elements in a combinationrange. The ease of user operation or operation efficiency can thereforebe greatly improved.

Further, even users who are not familiar with the details of effectssuch as the first-curtain synchronization effect can implement variousimage effects by selecting coefficient templates.

In the processing after step F220 shown in FIG. 22, similarly to FIG. 9,weighting coefficients can be individually changed as desired dependingon the image data elements or a combination range can be changed asdesired. Thus, the user can create a combined image with greateroriginality based on a combined image obtained by selecting acoefficient template.

After a coefficient template is selected, the coefficient patterndesignated by the coefficient template may be adjusted.

For example, with regard to the weighting coefficient pattern for themulti-flash effect, after the selection of the template, a lightemission interval (interval of highly weighted frames) may beadjustable. This adjustment operation may be implemented by adding thecorresponding function to one of the operation buttons or by selectingthe corresponding item from a menu. Alternatively, a light emissioninterval (interval of highly weighted frames) may be set as desired byperforming an operation such as pressing a left-right button whilepressing the shutter release button.

Furthermore, a coefficient template suitable for a specific use such asgolf swing check may be prepared.

A coefficient template may also be selected in a process other than thecombination process such as during image capture in the camera mode.

For example, when an image capture mode is selected in the camera mode,a mode selection screen as shown in FIG. 23 is displayed on the displaypanel 6.

In this case, in addition to the normal image capture modes such as theauto-mode image capture, portrait-mode image capture, sunset-scene-modeimage capture, and macro-mode image capture shown in FIG. 4, effectssuch as the long-time exposure effect, first-curtain synchronizationeffect, second-curtain synchronization effect, and multi-flash effectmay be selectable.

When one of the long-time exposure effect, first-curtain synchronizationeffect, second-curtain synchronization effect, and multi-flash effect isselected, in step F15 shown in FIG. 5, images of a plurality of framesare captured in the combination-mode image capture. During the imagecapture, information regarding the selected effect is recorded usingmetadata or the like for the image data group to be recorded.

Then, thereafter, such as when preview image combination is performed instep F203 in the combination process shown in FIG. 8 or when the CPU 31proceeds to step F206, weighted combination can be performed using thecoefficient template of the selected image effect.

For example, when the user wishes to achieve the first-curtainsynchronization effect during image capture, the coefficient templatefor the first-curtain synchronization effect is selected in advanceduring the image capture, thereby performing preview image combinationusing the weighting coefficient pattern for the first-curtainsynchronization effect. Thus, the operation efficiency can be improved.

Furthermore, in a case where the first-curtain synchronization effect,second-curtain synchronization effect, or multi-flash effect is selectedin this manner in the image capture stage, the CPU 31 may refer to theweighting coefficient pattern of the coefficient template to set atiming, and may actually perform flash light emission control at the settiming during the continuous frame image capture in step F15 shown inFIG. 5.

6. Image Capture Operation at Fixed Frame Rate

In an image capture apparatus such as a digital video camera or adigital still camera, the incident light fluxes entering image captureelements and the electric charge accumulation period are controlled toadjust the exposure time.

For example, in video cameras, generally, images are continuously shotat constant exposure periods according to a frame rate. However, ahigh-brightness subject may be captured using an electronic shutter thatprovides an exposure time shorter (for example, 1/250 seconds) than anexposure time defined by the frame rate (for example, about 1/60 secondsfor 60 frames per second (fps)).

In this case, no recording is performed for a time equal to a timeobtained by subtracting the actual exposure time of the electronicshutter from the exposure time defined by the frame rate. This mayresult in a moving image including jaggy. In order to prevent theproduction of such a moving image, exposure control is usuallyperformed. However, unless the moving image looks “jaggy”, image captureis generally performed using the electronic shutter for an electroniccharge accumulation time shorter than the exposure time defined by theframe rate.

Here, a consideration will be given of, as the combination-mode imagecapture, the capture of consecutive frames of image data in the imagecapture apparatus 1 of the present embodiment.

FIGS. 24B and 24D show the capture of consecutive frames at a fixedframe rate during image capture, where a period of one frame is FR.

FIG. 24B shows the case without using the electronic shutter function inthe frame periods FR, and an exposure period of image capture elementsis R1, where R1 is substantially equal to FR. That is, the image captureelements are continuously exposed to light within the frame periods,except for a minimum period necessary for electric charge transfercaused by the exposure. In the following description, the small amountof period during which the image capture elements are not exposed tolight due to the electric charge transfer is ignored for simplicity ofdescription.

FIG. 24D shows a situation in which automatic exposure adjustment isperformed using the electronic shutter function when it is necessary toreduce the amount of exposure because of the increase in the subjectbrightness. In this case, the exposure time within the frame period FRis shortened as indicated by R2. A period indicated by hatching is aperiod during which exposure is not performed.

FIGS. 24A and 24C show examples of a combined images obtained when, forexample, the continuous frame capture as shown in FIGS. 24B and 24D isperformed using the combination process described above. The combinedimages shown in FIGS. 24A and 24C are obtained when the long-timeexposure effect is achieved by assigning an equal weighting coefficientto image data of a plurality of frames.

A combined image that is based on image data captured without using theelectronic shutter function, as shown in FIG. 24B, results in a smoothlong-time exposure effect image, as shown in FIG. 24A.

On the other hand, a combined image that is based on image data capturedusing the electronic shutter function, as shown in FIG. 24D, results ina non-smooth long-time exposure effect image, as shown in FIG. 24C.

This is because information regarding subject images is not obtained forthe non-exposure periods indicated by hatching.

Here, an exposure control operation in a case where, as shown in FIG.24D, automatic exposure adjustment is performed using the electronicshutter function will be described with reference to FIGS. 25A to 25D.

The image capture apparatus 1 can perform exposure adjustment using theaperture mechanism and/or ND filter mechanism in the aperture/ND filtermechanism 22, the electronic shutter function executed by controllingthe operation of the image capture element unit 23 using the timinggeneration circuit 28, or variable gain to be applied by the analogsignal processing unit 24 to the image capture signal obtained by theimage capture element unit 23.

In FIGS. 25A and 25B, the abscissa axis represents the subjectbrightness level. FIG. 25A shows a period of one frame FR that isdefined by the frame rate during image capture. In this case, the frameperiod FR is set to 1/60 seconds, which is fixed regardless of thesubject brightness level.

FIG. 25B shows an electronic shutter SH (exposure time with a frameperiod or “intra-frame exposure time”) and a gain level G. FIG. 25Dschematically shows operation states of the aperture mechanism and theND filter mechanism.

FIG. 25C shows exposure control methods corresponding to changes in thesubject brightness level.

In a region having the lowest subject brightness level (region A),exposure adjustment is performed using the aperture mechanism.Specifically, the amount of incident light is adjusted by changing theamount of opening of the aperture mechanism.

In a region B where it is difficult to adjust the amount of incidentlight using only the aperture mechanism, both the aperture mechanism andthe ND filter mechanism are used. Specifically, the amount of incidentlight is adjusted by changing the amount of opening of the aperturemechanism and the amount of incident light flux entering the ND filtermechanism.

In a region C having too high subject brightness level to adjust theamount of incident light using only the aperture mechanism and the NDfilter mechanism, the electronic shutter function is used. For example,control is performed using the electronic shutter function so that theintra-frame exposure time SH, which is initially set to 1/60 seconds, isreduced as the subject brightness level is increased. For example, theintra-frame exposure time SH is controlled so as to be reduced down to1/500 seconds.

A region D is a brightness region having an extraordinarily high subjectbrightness level, where it is difficult to perform exposure adjustmenteven using the electronic shutter function. In the region D, the gainlevel G is changed.

When the frame rate is fixed, for example, the image capture apparatus 1may perform exposure adjustment control as described above. Then, theelectronic shutter function is used when the subject brightness level iswithin the region C. In this state, image capture is performed, therebycausing the situation shown in FIGS. 24C and 24D described above. Thus,a non-smooth combined image with the long-time exposure effect isobtained.

Accordingly, the image capture apparatus of the present embodiment isconfigured to perform exposure adjustment control as shown in FIGS. 26Ato 26D in a case where at least combination-mode image capture is to beperformed (that is, in a case where continuous frame capture in step F15shown in FIG. 5 is to be performed).

As in FIGS. 25A, 25B, 25C, and 25D, FIGS. 26A, 26B, 26C, and 26D showthe frame period FR (which is set to 1/60 seconds), the gain level G,the exposure time SH defined by an electronic shutter, exposure controlmethods, and operation states of the aperture mechanism and the NDfilter mechanism, where the abscissa axis represents the subjectbrightness level.

In a region A having the lowest subject brightness level, exposureadjustment is performed using the aperture mechanism. Specifically, theamount of incident light is adjusted by changing the amount of openingof the aperture mechanism.

In a region B where it is difficult to adjust the amount of incidentlight using only the aperture mechanism, both the aperture mechanism andthe ND filter mechanism are used. Specifically, the amount of incidentlight is adjusted by changing the amount of opening of the aperturemechanism and the amount of incident light flux entering the ND filtermechanism.

In a region C having too high subject brightness level to performadjustment using only the aperture mechanism and the ND filtermechanism, the adjustment is performed by variably controlling the gainlevel G without using the electronic shutter function.

Since gain adjustment may cause electronic noise to result in a capturedimage that lacks gradations or fineness, desirably, the gain adjustmentinvolves approaches such as setting a limit or performing a noisereduction process.

Only in a region D having an extraordinarily high subject brightnesslevel, which will not be addressed by changing the gain level up to thelimit, exposure adjustment is performed using the electronic shutterfunction.

In order to perform exposure adjustment as described above, in the imagecapture process in step F15 shown in FIG. 5 described above, the CPU 31(image capture control unit 51) performs an exposure adjustment processshown in FIG. 27. The process shown in FIG. 27 is continuously performedfor a period during which frames are continuously captured.

In step F301, the CPU 31 determines the subject brightness level. Forexample, the CPU 31 obtains exposure amount information (such as averagebrightness information) regarding the frame currently being captured andprocessed in the camera DSP 4, which is calculated by the image signalprocessing unit 41, and compares it with the exposure amount informationregarding the preceding frame to determine whether the subjectbrightness level has been increased or reduced.

If it is determined that no change has occurred in the subjectbrightness level, the CPU 31 returns to step F301 through steps F302 andF306 and determines the subject brightness level of a next frame.

If it is determined that the subject brightness level has beenincreased, the CPU 31 proceeds from step F302 to step F303 anddetermines whether or not the exposure adjustment method of the currentregion has reached its control limit. Specifically, it is checkedwhether or not the current increase of the subject light intensitycorresponds to one of the increases from the region A to the region B,from the region B to the region C, and from the region C to the region Dshown in FIG. 26C.

If the exposure adjustment method of the current region has not reachedits control limit, the CPU 31 proceeds to step F310 and addresses theincrease of the subject brightness level using the current exposureadjustment method.

Specifically, if the current region is the region A, the amount ofopening of the aperture mechanism is reduced. If the current region isthe region B, the amount of incident light is reduced using acombination of the aperture mechanism and the ND filter mechanism. Ifthe current region is the region C, control is performed so that thegain level is reduced. If the current region is the region D, control isperformed so that the intra-frame exposure time is shortened using theelectronic shutter function.

If the exposure adjustment method of the current region has reached itscontrol limit, the process branches in steps F304 and F305.

Specifically, if the subject brightness level has been increased withthe transition from the region A to the region B, the CPU 31 proceeds tostep F311. Then, the CPU 31 switches from the exposure adjustment usingonly the aperture mechanism to the exposure adjustment using both theaperture mechanism and the ND filter mechanism, and performs control toaddress the increase of the subject brightness level.

If the subject brightness level has been increased with the transitionfrom the region B to the region C, the CPU 31 proceeds to step F312.Then, the CPU 31 switches from the exposure adjustment using both theaperture mechanism and the ND filter mechanism to the exposureadjustment using variable gain levels, and performs control to addressthe increase of the subject brightness level.

If the subject brightness level has been increased with the transitionfrom the region C to the region D, the CPU 31 proceeds to step F313.Then, the CPU 31 switches from the exposure adjustment using variablegain levels to the exposure adjustment using the electronic shutterfunction, and performs control to address the increase of the subjectbrightness level.

If it is determined as a result of the determination in step F301 thatthe subject brightness level has been reduced, the CPU 31 proceeds fromstep F302 to step F306 and determines whether or not the exposureadjustment method of the current region has reached its control limit.Specifically, it is checked whether or not the current reduction of thesubject light intensity corresponds to one of the reductions from theregion B to the region A, from the region C to the region B, and fromthe region D to the region C shown in FIG. 26C.

If the exposure adjustment method of the current region has not reachedits control limit, the CPU 31 proceeds to step F310 and addresses thereduction of the subject brightness level using the current exposureadjustment method.

Specifically, if the current region is the region A, the amount ofopening of the aperture mechanism is increased. If the current region isthe region B, the amount of incident light is increased using acombination of the aperture mechanism and the ND filter mechanism. Ifthe current region is the region C, control is performed so that thegain level is increased. If the current region is the region D, controlis performed so that the intra-frame exposure time is increased usingthe electronic shutter function.

If the exposure adjustment method of the current region has reached itscontrol limit, the process branches in steps F308 and F309.

Specifically, if the subject brightness level has been reduced with thetransition from the region C to the region B, the CPU 31 proceeds tostep F311. Then, the CPU 31 switches from the exposure adjustment usingvariable gain levels to the exposure adjustment using both the aperturemechanism and the ND filter mechanism, and performs control to addressthe reduction of the subject brightness level.

If the subject brightness level has been reduced with the transitionfrom the region D to the region C, the CPU 31 proceeds to step F312.Then, the CPU 31 switches from the exposure adjustment using theelectronic shutter function to the exposure adjustment using variablegain levels, and performs control to address the reduction of thesubject brightness level.

If the subject brightness level has been reduced with the transitionfrom the region B to the region A, the CPU 31 proceeds to step F314.Then, the CPU 31 switches from the exposure adjustment using both theaperture mechanism and the ND filter mechanism to the exposureadjustment using only the aperture mechanism, and performs control toaddress the reduction of the subject brightness level.

The CPU 31 performs such processes to execute the exposure adjustmentshown in FIGS. 26A to 26D.

Specifically, the CPU 31 causes the image capture system 2 to capture aplurality of frames of image data having continuity in time at a fixedframe rate, and performs exposure adjustment control according to thesubject brightness level during the capture by performing exposureadjustment control using exposure adjustment functions (the aperturemechanism, the ND filter mechanism, and gain level control)preferentially to the electronic shutter function.

Such exposure adjustment control is performed in the image captureprocess in step F15 shown in FIG. 5, that is, the electronic shutterfunction is not used as much as possible. Thus, the image captureoperation as shown in FIG. 24B is realized in most cases.

In accordance with the subsequent combination process, as shown in FIG.24A, a smooth image, which is comparable to an image actually capturedusing long-time exposure, can be obtained. In particular, in the case ofcapturing an image of a moving subject, a smooth long-time exposureimage can be obtained.

This operation is performed on the assumption of, in particular, imagecapture at a fixed frame rate, which is suitable for a combination ofcombination-mode image capture and capture of a moving-image.

In a case where, for example, a plurality of frames are extracted frommoving-image data recorded as a result of the capture and are used in acombination process, the exposure adjustment operation described abovecan be performed to obtain image data satisfying both conditions ofappropriate moving-image data captured at a fixed frame rate and imagedata suitable for a combination process.

While the electronic shutter function is used in the region D, theregion D is a high-brightness region which hardly occurs in a usualcase, and the electronic shutter function will not be used in mostcases. Thus, a smooth combined image is obtained as a result ofcombination.

However, if continuous frame capture is performed in thecombination-mode image capture in step F15, the use of the electronicshutter function may be prohibited. That is, in the region D shown inFIG. 26C, exposure adjustment may be no longer performed or the gainlevel may be further reduced to perform the adjustment.

The exposure adjustment control described above may also be performed inthe image capture in step F16 as well in as step F15, or the exposureadjustment control shown in FIGS. 25A to 25D may be performed in stepF16.

While three exposure adjustment factors (the aperture mechanism, the NDfilter mechanism, and variable gain) other than the electronic shutterfunction are used in FIGS. 26 and 27, all the three factors may notnecessarily be used. For example, the ND filter mechanism may not beused.

In the image capture at a fixed frame rate, the following exemplaryimage capture operation can also be performed.

A plurality of frames of image data having continuity in time may becaptured at a fixed frame rate by, using the electronic shutterfunction, continuously executing divisional exposure within an exposureperiod defined by the fixed frame rate and combining image data of theplurality of frames obtained by the divisional exposure to produce imagedata of one frame.

This operation will now be described with reference to FIGS. 28A to 29C.

As described above, exposure adjustment control without using anelectronic shutter may cause electric charge accumulation in imagecapture elements such as CCD elements in excess of the limit thereof,resulting in saturation.

In this case, saturation of electric charge occurs in such a manner thatelectric charge overflows, and an overexposed image that is in a“whiteout condition” may be formed. Thus, a favorable image will not beobtained.

It is therefore difficult to return such an electric-charge-overflowingimage into the original state to obtain a favorable combined image.

In this case, for example, image capture may be performed using theelectronic shutter function so that the electric charge accumulated inimage capture elements such as CCD elements may not exceed a limit(saturation may not be occurred).

However, if the electronic shutter function is used in a usual manner, acombined image captured with a discontinuous exposure time as shown inFIGS. 28A and 28B may be obtained. For example, as shown in FIG. 28B, anexposure period R3 and a non-exposure period (hatched portion) exist ina period of one frame FR1. Due to the lack of information regarding thesubject in the non-exposure periods, a non-smooth combined image asshown in FIG. 28A may be obtained (which is similar to the case shown inFIGS. 24C and 24D described above).

Accordingly, as shown in FIG. 28D, in frame periods FR1 at a specificframe rate, for example, divisional exposure is performed in a mannerindicated by exposure periods R4 so that no non-exposure periods mayoccur.

In this case, in the image capture element unit 23, for example,electric charge is transferred every three exposure periods R4 intowhich the frame period FR1 is divided, and resulting captured image datais stored in, for example, the camera DSP 4. In the image captureelement unit 23, exposure resumes immediately after the transfer of theelectric charge and exposure in the exposure periods R4 is performed ina similar manner.

Then, three captured image data elements each corresponding to theexposure period R4 are combined in the camera DSP 4 to produce imagedata of one frame.

With this operation, captured image data of each frame (#1, #2, . . . ,) at a fixed frame rate is obtained. Furthermore, captured image datawith saturation removed in the image capture element unit 23 and havingno lack of information caused by non-exposure periods can be obtained.

Therefore, when the image data of the plurality of frames (#1, #2, . . .) is used in a combination process, a smooth long-time exposure effectimage shown in FIG. 28C can be obtained.

In this operation, the divisional exposure periods may be set in variousways. Each of the frame periods FR1 may be divided equally or unequally.In practice, adaptive control may be performed, taking the fixed framerate, the subject brightness level obtained during the image capture,the saturation time of electric charge in the image capture element unit23, or the like into account.

For example, in a case where the camera DSP 4 combines a plurality ofcaptured image data elements obtained using divisional exposure togenerate one frame of image data, it is desirable that weightedcombination be performed according to the divisional exposure time.

FIG. 29A shows divisional exposure in each exposure period R4, where aperiod of one frame FR1 is equally divided into three parts.

In this case, as shown in FIG. 29A, three captured image data elementsforming one frame are combined with the same weight (power of one) togenerate captured image data of respective frames (#0, #1, #2, . . . ).

FIG. 29B shows an example of divisional exposure in exposure periods R5,R6, and R7, where a period of one frame FR1 is unequally divided intothree parts. If the periods R5, R6, and R7 has a ratio of 3:2:1,combination is performed so that a weight of ⅓ is applied to the imagedata elements obtained in the exposure periods R5, a weight of ⅔ to theimage data elements obtained in the exposure periods R6, and a weight of1 to the image data elements obtained in the exposure periods R7.

FIG. 29C shows an example of divisional exposure in exposure periods R8and R9, where a period of one frame FR1 is partially unequally dividedinto three parts. If the periods R8 and R9 has a ratio of 3:1,combination is performed so that a weight of ⅓ is applied to the twoimage data elements obtained in the exposure periods R8 and a weight of1 to the image data element obtained in the exposure period R9.

7. Image Capture Operation at Variable Frame Rate

An exemplary image capture operation taking variable frame rates intoaccount will now be described.

For example, in the image capture in step F15 shown in FIG. 5, if theelectronic shutter function is used for automatic exposure adjustmentaccording to the subject brightness level, as described above, anon-smooth combined image is obtained when a combination process isperformed using a plurality of frames of image data obtained by theimage capture. FIGS. 30A and 30B show a combined image captured with adiscontinuous exposure in which, for example, a period of one frame isFR1 and exposure periods R3 within one frame is defined using theelectronic shutter function. When combination is performed using imagedata obtained using such image capture, a combined image as shown inFIG. 30A is obtained due to the lack of information in non-exposureperiods indicated by hatching.

Here, it is assumed that in the combination-mode image capture, a fixedframe rate is not necessarily used to obtain a plurality of consecutiveframes of image data used in a combination process. Then, image capturemay be performed in a manner shown in FIG. 30D. Specifically, a periodof one frame FR1 is not fixed, and the period of one frame is shortenedas the subject brightness level is increased (that is, variable framerates), and exposure is continuously performed for the period of oneframe.

In FIG. 30D, the frame rate is changed so that a period of one frame FR2can be equal to the exposure period R3, and an exposure adjustmenteffect equivalent to that shown in FIG. 30B is substantially achieved.Then, in FIG. 30D, because of no presence of non-exposure periods in theframe period FR2, image data of each frame is recorded without lack ofsubject information. Image data obtained by such image capture is usedto perform combination, resulting in a smooth long-time exposure effectcombined image shown in FIG. 30C.

Specifically, the CPU 31 (image capture control unit 51), when causingthe image capture system 2 to capture a plurality of frames of imagedata having continuity in time in step F15 shown in FIG. 5, performscontrol to change a frame rate in accordance with a change in thesubject brightness level during the capture. Then, the plurality offrames of image data having continuity in time captured by the imagecapture operation are used as combination-use image data to perform acombination process. Thus, a smooth long-time exposure effect image isobtained as a result of generating a combined image as a still image.

FIGS. 31A to 31C show an example of the relationship between changes inthe subject brightness level and frame rates.

It is now assumed that, as shown in FIG. 31A, the subject brightnesslevel is gradually increased starting from time t0 and becomessubstantially constant after time t1. In this case, automatic exposureadjustment is performed by changing the frame rate to control periods ofone frame in a manner shown in FIG. 31B. Specifically, exposureadjustment is performed so as to gradually shorten the periods of oneframe in order of FR1, FR2, FR3, . . . for the period of time t0 to timet1 during which the subject brightness level is gradually increased.Because of no presence of non-exposure periods in the period of oneframe, the one-frame exposure periods are shortened in order of R1, R2,R3, . . . in accordance with the length of the one frame period.

If no change occurs in the subject brightness level after time t1, forexample, the subsequent periods may be maintained at the frame periodFR8 (exposure period R8).

FIGS. 32A to 32D show an exemplary operation in a case where variableframe rate control as described above is used for automatic exposureadjustment.

In FIGS. 32A, 32B, 32C, and 32D, the abscissa axis presents the subjectbrightness level. FIG. 32A shows the frame cycle FR, FIG. 32B shows thegain level G, FIG. 32C shows exposure control methods, and FIG. 32Dshows operation states of the aperture mechanism and the ND filtermechanism.

Note that the electronic shutter function is not shown because it isassumed that exposure is performed throughout one period (except for ashort period for electric charge transfer) and the exposure time withina period of one frame is not shorter than the period of one frame.

In a region A having the lowest subject brightness level, exposureadjustment is performed using the aperture mechanism. Specifically, theamount of incident light is adjusted by changing the amount of openingof the aperture mechanism.

In a region B where it is difficult to adjust the amount of incidentlight using only the aperture mechanism, both the aperture mechanism andthe ND filter mechanism are used. Specifically, the amount of incidentlight is adjusted by changing the amount of opening of the aperturemechanism and the amount of incident light flux entering the ND filtermechanism.

In a region C having too high subject brightness level to performadjustment using only the aperture mechanism and the ND filtermechanism, the adjustment is performed by variably controlling the framerate. Specifically, a period of one frame FR is reduced as the subjectbrightness level is increased. For example, the frame period FR is setto 1/60 seconds when the subject brightness level is low, and iscontrolled to be variably reduced to 1/500 seconds as the subjectbrightness level is increased.

Only in a region D having an extraordinarily high subject brightnesslevel, which will not be addressed by changing the frame period FR to1/500 seconds, exposure adjustment is performed using variable gaincontrol.

In order to perform exposure adjustment as described above, in the imagecapture in step F15 shown in FIG. 5 described above, the CPU 31 (imagecapture control unit 51) performs an exposure adjustment process shownin FIG. 33. The process shown in FIG. 33 is continuously performed for aperiod during which frames are continuously captured.

Steps F301 to F311 and F314 are similar to those shown in FIG. 27described above. The processing of step F320 is performed instead ofF313 shown in FIG. 27, and the processing of step F321 is performedinstead of step F312 shown in FIG. 27. Since a portion of the specificprocessing steps is different such as step F310, all the processingsteps will be described hereinafter.

In step F301, the CPU 31 determines the subject brightness level. Forexample, the CPU 31 obtains exposure amount information (such as averagebrightness information) regarding the frame currently being captured andprocessed in the camera DSP 4, which is calculated by the image signalprocessing unit 41, and compares it with the exposure amount informationregarding the preceding frame to determine whether the subjectbrightness level has been increased or reduced.

If it is determined that no change has occurred in the subjectbrightness level, the CPU 31 returns to step F301 through steps F302 andF306 and determines the subject brightness level of a next frame.

If it is determined that the subject brightness level has beenincreased, the CPU 31 proceeds from step F302 to step F303 anddetermines whether or not the exposure adjustment method of the currentregion has reached its control limit. Specifically, it is checkedwhether or not the current increase of the subject light intensitycorresponds to one of the increases from the region A to the region B,from the region B to the region C, and from the region C to the region Dshown in FIG. 32C.

If the exposure adjustment method of the current region has not reachedits control limit, the CPU 31 proceeds to step F310 and addresses theincrease of the subject brightness level using the current exposureadjustment method.

Specifically, if the current region is the region A, the amount ofopening of the aperture mechanism is reduced. If the current region isthe region B, the amount of incident light is reduced using acombination of the aperture mechanism and the ND filter mechanism. Ifthe current region is the region C, control is performed so that theframe rate is changed to shorten the period of one frame. If the currentregion is the region D, control is performed so that the gain level isreduced.

If the exposure adjustment method of the current region has reached itscontrol limit, the process branches in steps F304 and F305.

Specifically, if the subject brightness level has been increased withthe transition from the region A to the region B, the CPU 31 proceeds tostep F311. Then, the CPU 31 switches from the exposure adjustment usingonly the aperture mechanism to the exposure adjustment using both theaperture mechanism and the ND filter mechanism, and performs control toaddress the increase of the subject brightness level.

If the subject brightness level has been increased with the transitionfrom the region B to the region C, the CPU 31 proceeds to step F321.Then, the CPU 31 switches from the exposure adjustment using both theaperture mechanism and the ND filter mechanism to the exposureadjustment using variable frame rates, and performs control to addressthe increase of the subject brightness level. If the subject brightnesslevel has been increased with the transition from the region C to theregion D, the CPU 31 proceeds to step F320. Then, the CPU 31 switchesfrom the exposure adjustment using variable frame rates to the exposureadjustment using variable gain levels, and performs control to addressthe increase of the subject brightness level.

If it is determined as a result of the determination in step F301 thatthe subject brightness level has been reduced, the CPU 31 proceeds fromstep F302 to step F306 and determines whether or not the exposureadjustment method of the current region has reached its control limit.Specifically, it is checked whether or not the current reduction of thesubject light intensity corresponds to one of the reductions from theregion B to the region A, from the region C to the region B, and fromthe region D to the region C shown in FIG. 32C.

If the exposure adjustment method of the current region has not reachedits control limit, the CPU 31 proceeds to step F310 and addresses thereduction of the subject brightness level using the current exposureadjustment method.

Specifically, if the current region is the region A, the amount ofopening of the aperture mechanism is increased. If the current region isthe region B, the amount of incident light is increased using acombination of the aperture mechanism and the ND filter mechanism. Ifthe current region is the region C, the period of one frame is increasedby changing the frame rate. If the current region is the region D,control is performed so that the gain level is increased.

If the exposure adjustment method of the current region has reached itscontrol limit, the process branches in steps F308 and F309.

Specifically, if the subject brightness level has been reduced with thetransition from the region C to the region B, the CPU 31 proceeds tostep F311. Then, the CPU 31 switches from the exposure adjustment usingvariable frame rates to the exposure adjustment using both the aperturemechanism and the ND filter mechanism, and performs control to addressthe reduction of the subject brightness level.

If the subject brightness level has been reduced with the transitionfrom the region D to the region C, the CPU 31 proceeds to step F321.Then, the CPU 31 switches from the exposure adjustment using variablegain levels to the exposure adjustment using variable frame rates, andperforms control to address the reduction of the subject brightnesslevel.

If the subject brightness level has been reduced with the transitionfrom the region B to the region A, the CPU 31 proceeds to step F314.Then, the CPU 31 switches from the exposure adjustment using both theaperture mechanism and the ND filter mechanism to the exposureadjustment using only the aperture mechanism, and performs control toaddress the reduction of the subject brightness level.

The CPU 31 performs such processes to execute the exposure adjustmentshown in FIGS. 32A to 32D.

Specifically, the CPU 31 causes the image capture system 2 to capture aplurality of frames of image data having continuity in time, andperforms exposure adjustment control according to the subject brightnesslevel during the capture by variably controlling the frame rate as wellas using the aperture mechanism, the ND filter mechanism, and gain levelcontrol to perform exposure adjustment control.

Such exposure adjustment control is performed in the image capture instep F15 shown in FIG. 5, whereby appropriate exposure adjustment can beperformed according to the subject brightness level and, in addition, aplurality of frames of image data captured has no lack of informationcaused by non-exposure periods.

Therefore, in accordance with the subsequent combination process, asshown in FIG. 30C, a smooth image, which is comparable to an imageactually captured using long-time exposure, can be obtained. Inparticular, in the case of capturing an image of a moving subject, asmooth long-time exposure image can be obtained.

In the region D, exposure adjustment is performed using variable gain.As described above, gain adjustment may cause electronic noise. However,the region D is a high-brightness region which hardly occurs in a usualcase, and exposure adjustment using variable gain will not be performedin most cases. Thus, it can be considered that there is substantially noeffect of noise caused variable gain in practice if a special techniquesuch as a noise reduction process is not used.

The exposure adjustment control described above may also be performed inthe image capture in step F16 as well as in step F15.

While three exposure adjustment factors (the aperture mechanism, the NDfilter mechanism, and the variable gain) other than the variable framerate are used in FIGS. 32 and 33, all the three factors may notnecessarily be used. For example, the ND filter mechanism may not beused.

In the combination process using image data elements obtained variablycontrolling the frame rate as described above, preferably, weightingcoefficients are assigned according to frame periods corresponding tothe image data elements. Specifically, weighting coefficients that aredetermined as the inverse of the ratio of frame period lengths of theimage data elements during the image capture are assigned to the imagedata elements used in the combining process to perform a combinationprocess.

For example, as shown in FIG. 31C, when image data elements #1 to #22obtained using variable frame rates are used as images to be combined toperform a combination process to generate combined-image data, weightingcoefficients shown in FIGS. 31A to 31C are assigned. For example, ifweighting coefficients to be assigned to the image data elements #8 to#22 in a frame period FR8 are 1, a weighting coefficient of ⅕ isassigned to the image data element #1, assuming that a frame period FR1corresponding to the image data element #1 is five times as long as theframe period FR8. A weighting coefficient of 3/10 is assigned to theimage data element #2, assuming that a frame period FR2 corresponding tothe image data element #2 is ( 10/3) times as long as the frame periodFR8.

In this manner, weighting coefficients that are determined as theinverse ratio are assigned to image data elements according to thelengths of the periods of one frame to generate combined-image data, anda smooth image can be produced based on the combined-image data.

In order to set such weighting coefficients in the combination process,desirably, information regarding frame rates is included in metadata tobe added to each image data element during image capture.

8. Exemplary Combination-Mode Process: Frame Interpolation

Various examples applicable in the combination-mode process describedabove will now be described. First, frame interpolation will bedescribed.

In the combination-mode image capture, images of a plurality of frameshaving continuity in time may be captured using the electronic shutterfunction for exposure adjustment or the like.

The electronic shutter function may be used in cases such as when theexposure adjustment method shown in FIGS. 25A to 25D is used, when asubject brightness level in the region D is obtained in the exposureadjustment method shown in FIG. 26C, and when a user performs thesetting of the electronic shutter function. In such cases, as describedabove, a non-exposure period occurs in a frame period even thoughcontinuous frame capture is performed, resulting in lack of subjectinformation the non-exposure period.

Further, even without using the electronic shutter function, forexample, in a case where the continuous image capture in step F15 shownin FIG. 5 is not performed every frame but is performed intermittentlysuch as every other frame, a sequence of a plurality of frames of imagedata having continuity in time also has lack of subject information.

In this situation, if a sequence of image data elements to be combinedhas lack of subject information, a combined image with non-smoothness isobtained.

However, a sequence of captured image data elements has no (orsubstantially no) lack of subject information if the exposure adjustmentmethod shown in FIGS. 26A to 26D or 32A to 32D described above is usedand if the continuous image capture in step F15 shown in FIG. 5 isperformed for all frames.

The sequence of image data elements can be used to obtain a combinedimage representing smooth motion of a subject. Another demand exists forimproved smoothness.

Therefore, preferably, a combination process is performed using a frameinterpolation process. The frame interpolation process will be describedwith reference to FIGS. 34A to 35B.

FIG. 34C shows a sequence of image data elements #1, #2, #3, and #4 thathave been actually captured and recorded on the recording medium 90 orthe like.

The image data elements #1, #2, #3, and #4 are combined, therebyobtaining a combined image shown in FIG. 34A. That is, a non-smoothimage with the long-time exposure effect is obtained.

Here, an interpolation process is performed to generate interpolatedframes #h12, #h23, and #h34 shown in FIG. 34D.

The interpolated frame #h12 is generated from the image data elements #1and #2. The interpolated frame #h23 is generated from the image dataelements #2 and #3. The interpolated frame #h34 is generated from theimage data elements #3 and #4.

An interpolated frame can be created, using an inter-frame (inter-field)interpolation technique generally used in codec schemes such as MPEGtechniques, by interpolating pixel values between two frames usingspatial prediction using motion vector.

FIG. 34B shows an image obtained by combining the image data elements#1, #h12, #2, #h23, #3, #h34, and #4. A smoother combined image can beachieved by performing combination using interpolated frames in additionto frames.

In the combination process using interpolated frames in addition toframes, weighting may be performed based on metadata (for example,exchangeable image file format (Exif) data) added to each of the imagedata elements #1, #2, #3, and #4 according to the image captureintervals or exposure times.

For example, FIG. 35A shows the image capture of the image data elements#1, #2, #3, and #4 for exposure periods R of 40 msec and non-exposureperiods (hatched portions) of 20 msec using the electronic shutterfunction, where a period of one frame FR1 is fixed.

In this case, the interpolated frames #h12, #h23, and #h34 serve tointerpolate frames for the periods of 20 msec, which have not beenactually captured. Frames that have been actually captured for theperiods of 40 msec with correct exposure can be used to create correctimages of interpolated frames.

In the combination process using the image data elements #1, #2, #3, and#4 and the interpolated frames #h12, #h23, and #h34, coefficients areassigned according to the ratio of exposure times.

For example, the image data element #1, the interpolated frame #h12, andthe image data element #2 have a relationship of 2:1:2 on the time axis.According to the computation of the inverse ratio, as shown in FIG. 35A,weights of 0.5:1:0.5 are assigned. Thus, exposure times can bereproduced in accordance with the actual motion of the subject.

The weights may be computed using a sum of brightness signals of aplurality of images or using exposure or shutter speed recorded in themetadata.

FIG. 35B shows image capture of the image data elements #1, #2, #3, and#4 without using non-exposure periods in a period of one frame FR2.

Because of no presence of non-exposure periods, the interpolated frames#h12, #h23, and #h34 are used for the purpose of obtaining a more smoothcombined image. In this case, as shown in FIG. 35B, a combinationprocess is performed by assigning an equal weight to the image dataelements #1, #2, #3, and #4 and the interpolated frames #h12, #h23, and#h34.

FIG. 36 shows an exemplary process of the CPU 31, including the frameinterpolation process described above.

For example, in the adjustment process in step F206 shown in FIG. 8, theCPU 31 performs a process shown in FIG. 36 instead of the example shownin FIG. 9.

In step F206 shown in FIG. 8, the CPU 31 performs the process shown inFIG. 36. First, in step F220, as in FIG. 9, the CPU 31 displays aselection image list and weight bars. For example, the combination-workimage 70 is changed from the state shown in FIG. 12 to the state shownin FIG. 13.

At this time, a combined image to be displayed as a preview image in theimage display area 72 is an image generated using the frameinterpolation process in steps F230 and F231.

Specifically, in step F230, the CPU 31 creates interpolated framesdescribed with reference to FIG. 34D using a sequence of image dataelements in the current combination range. In step F231, the CPU 31 setspredetermined weighting coefficients in a manner described withreference to FIG. 35A or 35B to perform a combination process. Thus, acombined image displayed in the image display area 72 of thecombination-work image 70 shown in FIG. 13 is, for example, a combinedimage obtained by performing a combination process together with a frameinterpolation process.

The subsequent processing of steps F221 to F225A shown in FIG. 36 issimilar to that shown in FIG. 19, and redundant descriptions thereof areomitted. However, when preview image combination is performed in stepsF224A and F225A, the combination is performed using the interpolatedframes generated in step F230.

Further, when a final combination process is performed in step F207shown in FIG. 8 after the adjustment process shown in FIG. 36 isperformed, the combination process is performed also using theinterpolated frames used in the final stage of the adjustment process instep F206 (FIG. 36).

With the additional use of the frame interpolation process, a smootherimage, which is comparable to an image actually captured using long-timeexposure, can be obtained as a combined image.

In particular, when a rapidly moving subject is a target, moreadvantageously, smooth motion in a combined image can be representedusing the frame interpolation process described above.

In the example described above, in the process shown in FIG. 36 in stepF206 shown in FIG. 8, a combined image necessarily using interpolatedframes is displayed as a preview image. However, in this adjustmentprocess, image combination involving the generation and use ofinterpolated frames may be performed only when it is desired by a useroperation.

Further, frame interpolation may not necessarily be performed in theadjustment process in step F206 shown in FIG. 8. Instead, in the finalcombination process in step F207, interpolated frames may be generatedand used to perform image combination.

Furthermore, in the adjustment process, in the combination-work image70, interpolated frames may be displayed on a combined image listtogether with original image data, as well as weight barscorrespondingly, so as to be used to change weighting coefficients in amanner similar to that in the original image data or to set acombination start position/combination end position.

Alternatively, conversely, the presence of interpolated frames may behidden from the user.

9. Exemplary Combination-Mode Processes: Flash Removal/Correction

A user may perform the combination-mode image capture (image capture instep F15 shown in FIG. 5) using a flash. In accordance with thesubsequent combination process, for example, the first-curtainsynchronization effect or the like can be achieved without using aflash. However, some users may perform image capture using a flash incases such as when they wish to initially achieve the first-curtainsynchronization effect or the like without changing weightingcoefficients or the like in the combination work or to obtain an imagerepresentation with the use of a flash.

In such cases, however, it is necessary to determine settings accordingto photographer's experience and intuition, such as the flash lightemission time and the amount of exposure. In practice, a user(photographer) may not necessarily obtain the desired image.

Further, after a user actually captures an image using a flash, the usermay later change his or her mind and wish to generate a combined imageusing images captured without using a flash.

Moreover, the actual amount of flash may be insufficient due to theperformance of the image capture apparatus or the user's settings.

It is therefore desirable for a user to be able to remove or correctflash image capture in the combination process.

FIG. 38 shows an exemplary process of the CPU 31 for enabling flashremoval in the adjustment process in step F206 in the combinationprocess shown in FIG. 8.

Steps F220 to F225A shown in FIG. 38 are similar to those in FIG. 19. InFIG. 38, steps F240 to F245 are added to the process shown in FIG. 19.

When the combination-work image 70 is being displayed in the adjustmentprocess, in addition to the operation of changing a weightingcoefficient or changing a combination range described above, a user canperform a flash removal operation. The flash removal operation can beperformed by, for example, operating a predetermined operation key orselecting a menu item.

When a flash removal operation performed by the user is detected, theCPU 31 advances the process from step F240 to step F241.

In steps F241 to F244, a weighting coefficient for removing the flash ischanged for each image data element within a combination range.

In step F241, the CPU 31 extracts one of the image data elements in thecombination range as a processing target. In step F242, the CPU 31determines whether the extracted image data element is an image dataelement captured using a flash (hereinafter referred to as a “flashimage”) or an image data element captured without using a flash(hereinafter referred to as a “non-flash image”). As described above,the determination of a flash image or a non-flash image may be performedby checking the metadata added to image data element during imagecapture.

In step F243, the CPU 31 changes the weighting coefficient depending ona flash image or a non-flash image.

Specifically, if the current image data element is a flash image, theweighting coefficient is reduced so as to remove or negate the flasheffect. If the current image data element is a non-flash image, theweighting coefficient is increased.

The weighting coefficients are used for the purpose of eliminating adifference in overall image brightness level between a flash image and anon-flash image. Thus, the amount by which the weighting coefficient fora flash image is reduced and the amount by which the weightingcoefficient for a non-flash image is increased are determined accordingto the relative difference in overall image brightness leveltherebetween. In some cases, when the weighting coefficient of the flashimage is reduced while the weighting coefficient of the non-flash imageis maintained unchanged, the overall brightness level of the flash imagemay be equivalent to that of the non-flash image.

After the process has been completed for one image data element, the CPU31 returns from step F244 to step F241 and performs a similar process ona next image data element in the combination range.

When the process described above is completed for all the image dataelements in the combination range, the weighting coefficients set forthe individual image data elements are used to equalize the differencesin overall screen brightness level between the individual image dataelements. That is, the weighting coefficients are assigned to theindividual image data elements so that flash images can become as brightas non-flash images.

In this state, in step F245, the CPU 31 changes the heights of theweight bars corresponding to the image data elements. The CPU 31 furtherperforms a combination process using the weighting coefficientscorresponding to the image data elements to generate a combined imagewith the flash removed, and displays the combined image in the imagedisplay area 72 as a preview image.

The image that has been displayed in the image display area 72 (that is,the previous combined image) is further displayed in the image displayarea 71 as a preview image of a combined image before the flash removal.

FIG. 37 shows a combination-work image 70 obtained after the flashremoval process.

It is assumed that, as shown in FIG. 37, an image data element #11 is aflash image among image data elements #5 to #11 in a combination rangethat are displayed as a selection image list in the timeline 73.

Initially, when the adjustment process shown in FIG. 38 is started, thatis, when an equal weighting coefficient is assigned to the image dataelements #5 to #11 (for example, as shown in FIG. 13, in an initialstate where the weighting coefficients are equal and where the weightbars have the same height), the combined image with the second-curtainsynchronization effect that is displayed in the image display area 71shown in FIG. 37 is displayed in the image display area 72 as a previewimage of the combined image.

Then, when the user performs the flash removal operation to perform theprocessing of steps F241 to F245, the combination-work image 70 shown inFIG. 37 is obtained.

Specifically, the weighting coefficients of the image data elements #5to #10, which are non-flash images, are increased, which is representedby increasing the weight bars w5 to w10. Further, the weightingcoefficient of the image data element #11, which is a flash image, isreduced, which is represented by reducing the weight bar w11.

Then, the combined image obtained after the flash removal process isdisplayed in the image display area 72, and the combined image obtainedbefore the flash removal is displayed in the image display area 71.

As shown in FIG. 37, the combined image obtained after the flash removalprocess is a long-time exposure effect image with the second-curtainsynchronization effect removed. That is, a combined image is generatedso that the overall brightness level of the image data element #11,which is a flash image, can also be equal to that for a non-flash image.

In this manner, even when a user captures images using a flash, the usercan easily remove the flash effect as desired in the combinationprocess.

For example, when a user obtains an undesired image as a result of flashimage capture, the user can only perform the flash removal operation toassign a weight for a non-flash image to obtain a combined image.

While the determination of a flash image or a non-flash image in stepF242 may be performed using, instead of metadata, a brightnessdistribution (sum of brightness signals) of image data of preceding andfollowing frames in consecutive frames or the like.

For example, a sum of brightness signals for consecutive captured imagescan be detected, monitored, or watched to detect a change in thebrightness signals.

The determination of a frame captured with the firing of the flash canalso be performed, although indirectly, by detecting a light controlfunction (such as the aperture mechanism, the ND filter mechanism, or aliquid crystal iris) in exposure control, the shutter speed, theelectronic shutter speed, the gain, or the like.

When a rapid change is found by detecting the rate of overtime changesof a brightness signal (or lightness of a scene of a captured image), itis determined that a flash has been fired. For example, when a changefrom a bright condition to a dark condition within a period of 1/60seconds is found, it can be determined that a flash has been fired.

In this manner, for example, with the determination based on brightnesscomparison between frames, exposure control conditions, or otherconditions, the use or non-use of a flash can be determined in casesother than flash light emission by the image capture apparatus 1, suchas when a flash was fired from another device close to the image captureapparatus 1 or when a subject was instantaneously illuminated by theheadlight of a vehicle passing by.

Therefore, for example, assuming that flash light emission caused byuser's unintentional events in a situation where a large number ofphotographers are close to the user is to be removed or negated in thecombination process, it is preferable that a flash image or a non-flashimage be determined based on the brightness comparison between frames,exposure control conditions, or the like. That is, regardless of theimage capture conditions, a combined image can be easily obtained withthe ambient image capture conditions canceled.

An exemplary process for enabling flash correction will now be describedwith reference to FIGS. 39 and 40.

In the case of an insufficient amount of flash even though the usercaptures images using a flash, an undesired image effect combined imagemay be obtained in the combination process.

Accordingly, it is desirable to correct a flash image in the combinationprocess.

FIG. 40 shows an exemplary process of the CPU 31 for enabling flashcorrection in the adjustment process in step F206 in the combinationprocess shown in FIG. 8.

Steps F220 to F225A shown in FIG. 38 are similar to those in FIG. 19. InFIG. 38, steps F250 to F255 are added to the process shown in FIG. 19.

When the combination-work image 70 is being displayed in the adjustmentprocess, in addition to the operation of changing a weightingcoefficient or changing a combination range described above, a user canperform a flash correction operation. The flash correction operation canbe performed by, for example, operating a predetermined operation key orselecting a menu item.

When a flash correction operation performed by the user is detected, theCPU 31 advances the process from step F250 to step F251.

In steps F251 to F255, a flash image is extracted among image dataelements in a combination range, and the weighting coefficient ischanged.

In step F251, the CPU 31 extracts one of the image data elements in thecombination range as a processing target. In step F252, the CPU 31determines whether the extracted image data element is a flash image ora non-flash image. As described above, the determination of a flashimage or a non-flash image may be performed by checking the metadataadded to image data element during image capture. Alternatively, asdescribed above, the determination may be based on the brightnesscomparison between frames, exposure control conditions, or the like.

If the current target image data element is a flash image, the CPU 31proceeds to step F253 and changes a weighting coefficient. In this case,the frame of the flash image is strongly weighted to perform correctionso that the flash has been emitted with an appropriate light amount.

If the current target image data element is a non-flash image, theweighting coefficient is not specifically changed.

After the process has been completed for one image data element, the CPU31 returns from step F254 to step F251 and performs a similar process ona next image data element in the combination range.

When the process described above is completed for all the image dataelements in the combination range, the weighting coefficients set forthe individual image data elements are used to perform correction sothat brightness level of a flash image can be improved.

In this example, the weighting coefficient of a non-flash image ischanged. However, the weighting coefficient of a non-flash image may bechanged so as to be reduced or the like to highlighted a flash image.

After the processing of steps F251 to F254 has been completed for allthe image data elements in the combination range, in step F255, the CPU31 changes the heights of the weight bars corresponding to the imagedata elements. The CPU 31 further combines the individual image dataelements using the changed weighting coefficients to generate a combinedimage with the flash corrected, and displays the combined image in theimage display area 72 as a preview image.

A combined image that has not been corrected for the flash effect (thatis, the image that has been displayed in the image display area 72 as apreview image of the combined image immediately before the flashcorrection process) is further displayed in the image display area 71.

FIG. 39 shows a combination-work image 70 obtained after the flashcorrection.

It is assumed that, as shown in FIG. 39, an image data element #11 is aflash image among image data elements #5 to #11 in a combination rangethat are displayed as a selection image list in the timeline 73.

Initially, when the adjustment process shown in FIG. 40 is started, thatis, when an equal weighting coefficient is assigned to the image dataelements #5 to #11 (for example, as shown in FIG. 13, in an initialstate where the weighting coefficients are equal and where the weightbars have the same height), the combined image with the second-curtainsynchronization effect that is displayed in the image display area 71shown in FIG. 39 is displayed in the image display area 72 as a previewimage of the combined image. In this case, due to the insufficientamount of flash during the image capture, the overall brightness levelof the image data element #11 is low and the second-curtainsynchronization effect is not significantly highlighted.

Then, when the user performs the flash correction operation to performthe processing of steps F251 to F255, the combination-work image 70shown in FIG. 39 is obtained.

Specifically, the weighting coefficient of the image data element #11,which is a flash image, is increased, which is represented by increasingthe weight bar w11.

Then, the combined image obtained after the flash correction process isdisplayed in the image display area 72, and the combined image obtainedbefore the flash correction is displayed in the image display area 71.

As shown in FIG. 39, the combined image obtained after the flashcorrection process is an image in which the second-curtainsynchronization effect is noticeable.

In this manner, even when a user captures images using a flash but thesubject brightness level of the images is not sufficient due to theperformance of the flash, the distance to a subject, or the like, theuser can easily correct the flash effect as desired in the combinationprocess.

In this embodiment, flash correction has been described in the contextof the achievement of a sufficient image brightness level in the case ofinsufficient amount of light. Conversely, when the amount of flash istoo large, correction may be performed so that the screen intensity canbe reduced.

Further, the amount by which the weighting coefficient for a flash imageis corrected may be automatically calculated by the CPU 31 or may beadjusted step-by-step by the user.

In the case of automatic calculation, for example, a standard differencebetween total sums of brightness levels of flash images and non-flashimages may be set in advance and weighting coefficients of the flash andnon-flash images may be adjusted so that the difference can be obtained.

In the case of adjustment by the user, a fixed amount of change may beset for the weighting coefficients, and the weighting coefficient for aflash image may be changed by the fixed amount each time the flashcorrection operation is performed. The user merely repeats the flashcorrection operation until a desired state has been obtained.

Since the flash removal and flash correction processes described aboveare implemented by changing a weighting coefficient, the user can alsothe processes by selecting an image data element in the combination-workimage 70 and manually changing the weighting coefficient. However, it isactually time-consuming for a user to change individual weightingcoefficients of a large number of image data elements. By executing theprocesses described above in accordance with a flash removal operation,flash correction operation, or the like, the operability of thecombination work is significantly improved.

10. Exemplary Combination-Mode Processes: Distance-Based Correction

An exemplary process for correcting a weighting coefficient according tothe distance to a subject during image capture will now be describedwith reference to FIGS. 41 to 44.

In order to perform this process, in the image capture in step F15 shownin FIG. 5, a distant to the in-focus plane during the image capture(hereinafter referred to as an “image capture distance”) is included inmetadata (for example, Exif data) added to image data of each frame. Asdescribed above, a main subject is in focus using autofocus controlduring image capture. At this time, the lens position detection unit 27performs the inverse operation from the lens address to determine thedistance to the main subject. The CPU-31 adds information regarding thedistance as metadata.

Now, in the adjustment process in step F206 shown in FIG. 8 in thecombination process, it is assumed that image data elements #5 to #11displayed as a selection image list in the timeline 73 shown in FIG. 41are in a combination range.

It is assumed that the image data elements #5 to #11 show a scene inwhich a subject is gradually approaching the image capture apparatus 1while moving from the upper right to the lower left of the screen.

A combined image generated with the same weighting coefficient assignedto the image data elements #5 to #11 is displayed as a preview image inthe image display area 72.

Here, it is assumed that, as shown in FIG. 42, a user performs aweighting coefficient operation to increase the weighting coefficientfor the image data elements #5, #8, and #11 and reduce the weightingcoefficient for the remaining image data elements #6, #7, #9, and #10 toapply the multi-flash effect so as to produce an image displayed as apreview image.

Since the image data elements #5 to #11 have been captured without usinga flash, the multi-flash effect is achieved using the weightingcoefficients. Here, the actual firing of the flash during image capturewill be considered.

In flash image capture, actually, as the distance to the subjectincreases, the amount of light reaching the subject is reduced,resulting in a dark shot if the amount of light is not sufficient. Inorder to avoid this problem, a larger flash is necessary, which mayprevent reduction in size and power consumption.

In the present embodiment, in contrast, the problem caused by the actualfiring of the flash in the related art can be overcome by performingimage combination as shown in FIG. 42. That is, an image similar to thatcaptured with a sufficient amount of light can be obtained without usinga large flash.

There may be another demand for an image similar to that captured withthe firing of the flash to be reproduced more realistically withoutusing a flash. That is, a demand may be considered for a more realisticreproduction of the multi-flash effect that is achieved using the flashimage capture of the related art.

Therefore, it is desirable to correct the flash effect. FIG. 43 shows acombined image with the multi-flash effect, which is similar to an imageactually captured with the firing of the flash, in which the weightingcoefficients of image data elements (#5, #8, and #11) with the flasheffect applied are increased according to information regarding theimage capture distance recorded in metadata (for example, Exif data)while adjusting the weighting coefficients depending on the imagecapture distance (hereinafter referred to as “distance-based flashcorrection”). Specifically, in FIG. 43, as indicated by weight bars w5,w8, and w11, the weighting coefficients are adjusted so as to be reducedas the image capture distance increases. Thus, as shown in a previewimage in the image display area 72 which is obtained after thedistance-based flash correction, a combined image is produced so thatthe flash effect is diminished as the distance increases. The combinedimage obtained before the distance-based flash correction (that is, thecombined image obtained in the state shown in FIG. 42) is displayed inthe image display area 71. As can be seen from the comparison betweenboth combined images, a more realistic representation of a combinedimage can be achieved using the distance-based flash correction.

FIG. 44 shows a process of the CPU 31 for implementing thedistance-based flash correction described above. FIG. 44 shows anexemplary process for enabling distance-based flash correction in theadjustment process in step F206 in the combination process shown in FIG.8.

Steps F220 to F225 shown in FIG. 44 are similar to those shown in FIG.9. In FIG. 44, steps F260 to F266 are added to the process shown in FIG.9.

When the combination-work image 70 is being shown in the adjustmentprocess, as described above, in addition to the operation of changing aweighting coefficient or changing a combination range, a user canperform a distance-based flash correction operation. The flashcorrection operation can be performed by, for example, operating apredetermined operation key or selecting a menu item.

For example, the user may perform a weighting coefficient operation (orselect the coefficient template for the multi-flash effect) to apply themulti-flash effect to a combined image in the manner shown in FIG. 42.The multi-flash effect is merely an example, and any other effect suchas the first-curtain synchronization effect, the second-curtainsynchronization effect, or the flash effect with a high weightingcoefficient assigned to a certain image data element in a combinationrange may be applied.

In this case, the user can achieve a more realistic image representationby performing the distance-based flash correction operation.

When a distance-based flash correction operation performed by the useris detected, the CPU 31 advances the process from step F260 to stepF261.

In steps F261 to F264, an image with the flash effect applied using aweighting coefficient is extracted among image data elements in acombination range, and the weighting coefficient is changed.

In step F261, the CPU 31 extracts one of the image data elements in thecombination range as a processing target. In step F262, the CPU 31determines whether or not the extracted image data element is an imagewith the flash effect applied using the weighting coefficient(hereinafter referred to as a “flash-effect-applied image” or“flash-effect-applied image data element”).

If the current target image data element is a flash-effect-appliedimage, the CPU 31 proceeds to step F263 and determines the image capturedistance of the image data element with respect to the subject on thebasis of the metadata of the image data element. Then, in step F264, theCPU 31 calculates a correction value of the weighting coefficient on thebasis of the image capture distance, and stores it as the correctionvalue of the weighting coefficient of the image data element.

If the current target image data element is not a flash-effect-appliedimage, the processing of steps F263 or F264 is not performed.

After the process has been completed for one image data element, the CPU31 returns from step F265 to step F261 and performs a similar process ona next image data element in the combination range.

When the process described above is completed for all the image dataelements in the combination range, the correction values of theweighting coefficients based on the image capture distances have beencalculated for all the flash-effect-applied image data elements in thecombination range (in FIG. 42, the image data elements #5, #8, and #11).

After the processing of steps F261 to F265 has been completed for allthe image data elements in the combination range, in step F266, the CPU31 corrects the weighting coefficients of the flash-effect-applied imagedata elements on the basis of the correction values of the weightingcoefficients based on the image capture distances.

After the correction, the CPU 31 changes the heights of the weight barscorresponding to the image data elements. The CPU 31 further combinesthe individual image data elements using the corrected weightingcoefficients to generate a combined image, and displays the combinedimage in the image display area 72.

A combined image obtained before the distance-based flash correction(that is, the image that has been displayed in the image display area 72as a preview image of the combined image immediately before the process)is further displayed in the image display area 71.

With the process described above, the combination-work image 70 shown inFIG. 43 is obtained. Thus, a combined image subjected to distance-basedflash correction is displayed.

In the combination in step F266, in order to prevent the lightness ofthe combined image generated after the correction of weightingcoefficients from being entirely changed, all the weighting coefficientsmay be automatically adjusted so as to obtain an appropriately brightcombined image.

In the process shown in FIG. 44, the weighting coefficient of only aflash-effect-applied image data element is corrected according to thedistance. Alternatively, correction values of the weighting coefficientsof all the image data elements in the combination range may becalculated according to the image capture distances and the weightingcoefficients may be corrected. Specifically, the processing of step F262shown in FIG. 44 may be omitted, and the processing of steps F263 andF264 may be performed for all the image data elements in the combinationrange.

In this case, a combination-work image 70 shown in FIG. 45 is obtainedafter distance-based correction. Specifically, as indicated by theweight bars w5 to w11, the weighting coefficients of the image dataelements #5 to #11 are corrected according to the image capturedistances, and a combined image based on the corrected weightingcoefficients is displayed in the image display area 72.

In actuality, the flash reach distance of a flash fired has thefollowing relationship:Flash reach distance (m)=guide number (ISO 100)÷aperture setting value

Once a guide number is specified, a correction value of a weightingcoefficient can be set based on the fact that the amount of lash lightis in inverse proportion to the square of the distance.

In view of image representations, preferably, a virtual guide number isspecified in accordance with an instruction given by a user operation. Aguide number may also be automatically selected in accordance with thedistance distribution of subjects. For example, when subjects aredistributed over a short distance to a middle distance, the guide numberis reduced; when subjects are distributed over a short distance to along distance, the guide number is increased.

A guide number may also be preset such as “a guide number of 28”. The“guide number” is merely an example and any other value that implies theamount of flash light may be specified such as a flash reach distance.Or, an aperture setting value different from the aperture value used inthe actual photographing may be specified. Thus, a virtual flash effectis achieved.

When a coefficient template is selected in the manner described above,weighting can also be determined by referring to metadata of capturedimage data on the basis of the image capture distance.

The multi-flash effect may be automatically set so that light emissionintervals (intervals of highly weighted frames) can be adjusted after acoefficient template is selected and so that weighting can be determinedbased on the image capture distance according to the adjusted lightemission intervals.

The adjustment operation may be implemented by adding the correspondingoperation function to a predetermined operator or by selecting thecorresponding item from a menu. Alternatively, a light emission intervalmay be set as desired by performing an operation such as pressing aleft-right button of the cross key 5 i while pressing the shutteroperation key 5 a.

Further, area extraction may be used to apply weighting only to anin-focus subject according to the image capture distance, therebyachieving a more realistic reproduction of the flash effect.

Distance distribution information such as the so-called depth map (ameasured distance to a subject for each pixel) may be obtained asinformation regarding the image capture distances, thereby assigning aweight to each pixel according to the relationship of:Flash reach distance (m)=guide number (ISO 100)÷aperture setting value

Specifically, in addition to using weighting to obtain a long-timeexposure image during image combination, weights may be changeddepending on distances within one combined image using area extractionbased on the image capture distance or distance distribution informationsuch as a depth map, thereby achieving a more realistic reproduction ofthe flash effect.

As described above, the distance distribution information can begenerated by the information generation unit 44 of the camera DSP 4during image capture.

Further, a frame having a short image capture distance may be stronglyweighted and a frame having a long image capture distance may be weaklyweighted to reduce an image processing load. The processing load can bereduced by performing weighting on the entire screen without using areaextraction or a depth map.

In an image captured using long-time exposure for representing motion ofa moving subject, no motion may be contained in the background areas oreven motion contained in the background areas may not be necessarilyimportant in terms of image capture. Thus, in some scenes captured, noproblems occur if image combination is performed by performing weightingon the entire screen.

According to the exemplary process described above with reference toFIGS. 41 to 45, weighting for image combination is performed accordingto subject distance information. Thus, an image similar to that actuallycaptured using a flash can be obtained without using a flash. Therefore,a more realistic representation of a combined image with the multi-flasheffect, the first-curtain synchronization effect, the second-curtainsynchronization effect, the light emission effect at a desired time, orthe like can be achieved.

Another exemplary process will now be described with reference to FIGS.46 and 47.

For example, it is assumed that image data elements #5, #8, and #11among image data elements #5 to #11 in a combination range shown in FIG.46 have been actually captured using a flash.

Further, due to the sufficient amount of flash light, it is assumed thata favorable, appropriate exposure image of a frame having a long imagecapture distance is not obtained. In this example, it is assumed thatthe amount of flash light for the image data element #11 in which thesubject is close to the image capture apparatus 1 is sufficient whilethe amount of light for the image data elements #8 and #5 in which thesubject is distant is not sufficient (the amount of light for the imagedata element #5 is the smallest).

In this case, when combination is performed by applying an equalweighting coefficient, as shown in FIG. 46, an image with themulti-flash effect in which the subject is more noticeable when it iscloser to the image capture apparatus 1 is obtained.

Although this combined image provides a realistic representation of themulti-flash effect, some users may desire a clear, rather thanrealistic, representation of the light emission effect regardless of theimage capture distance. Therefore, correction may be performed so that,conversely to the example described above, a realistic image obtained byactually performing image capture with flash light emission can becorrected to enhance the flash effect.

In order to perform this process, in the image capture in step F15 shownin FIG. 5, an image capture distance (a distance to the in-focus planeduring the image capture) is included in metadata (for example, Exifdata) to be added to each frame of image data. Information as to whetheror not a flash has been emitted is also included as metadata.

An actual exemplary process of the CPU 31 may be implemented bymodifying the process shown in FIG. 44 described above.

Specifically, in step F262, it is determined whether or not theextracted image data element is a flash image that has been captured byactually firing a flash. If the extracted image data element is a flashimage, in step F263, a distance to the subject is determined. In stepF264, a correction value of the weighting coefficient is calculatedaccording to the distance. At this time, conversely to the exampledescribed above, a correction value for correcting the weightingcoefficient so as to provide a sufficient amount of light, instead ofreducing the amount of flash light according to the distance, isdetermined.

Then, after the correction values of the weighting coefficients of allflash images are calculated, in step F266, the weighting coefficients ofthe flash images may be corrected using the correction values, and,after the correction, a combination process may be performed.

FIG. 47 shows an example of the combination-work image 70 obtained afterthe correction. The corrected combined image is displayed in the imagedisplay area 72. In the combined image obtained after the correction, ascan be seen from the comparison with the combined image obtained beforethe correction, a clearer representation of the flash effect can beobtained even for an image actually captured with an insufficient amountof flash light for the subject which is distant.

As indicated by weight bars w5 to w11, the weighting coefficients arecorrected so that a higher weighting coefficient is assigned to an imagedata element in which the subject is more distant among the flash images#5, #8, and #11. That is, correction is performed so as to provide thefiring of the flash with a large amount of light.

In the combination in step F266, in order to prevent the lightness ofthe combined image generated after the correction of weightingcoefficients from being entirely changed, all the weighting coefficientsmay be automatically adjusted so as to obtain an appropriately brightcombined image.

In this case, the correction of a weighting coefficient of a flash image(the calculation of a correction value in step F264) can be performed asfollows.

It is determined whether or not it is necessary to correct a flash imagecaptured using a flash (it is necessary to change the weightingcoefficient) according to the following relationship:Flash reach distance (m)=guide number (ISO 100)÷aperture setting value

For example, if the image capture distance (m) is smaller than the flashreach distance (m), it is determined that the amount of flash light isinsufficient and that the correction (weighting) is necessary.

If the correction is necessary, correction is performed based on thefact that the amount of flash light is in inverse proportion to thesquare of the distance and the relationship that “flash reach distance(m)=guide number (ISO 100)÷aperture setting value”.

In this correction, a guide number is specified to perform weightingbased on the fact that the amount of flash light is in inverseproportion to the square of the distance.

In view of image representations, preferably, a virtual guide number isspecified in accordance with an instruction given by a user operation. Aguide number may also be automatically selected in accordance with thedistance distribution of subjects. For example, when subjects aredistributed over a short distance to a middle distance, the guide numberis reduced; when subjects are distributed over a short distance to along distance, the guide number is increased.

A guide number may also be preset such as “a guide number of 28”.

The “guide number” is merely an example and any other value that impliesthe amount of flash light may be specified such as a flash reachdistance. Or, an aperture setting value different from the aperturevalue used in the actual image capture may be specified. Thus, a virtualflash effect is achieved.

The following process may be used as an exemplary weighting coefficientcorrection process.

The degree of correction (weighting) necessary for a frame captured withthe firing of the flash (flash image) may be determined by determining acorrection coefficient according to the brightness distribution of theimage and performing correction (weighting) using the metadata and thecorrection coefficient.

Further, the determination of a frame captured with the firing of theflash (flash image) may be based on the brightness distribution (sum ofbrightness signals) or the like of preceding and following frames ofconsecutive images continuously captured and recorded.

Moreover, the determination of a frame captured with the firing of theflash (flash image) may be performed by referring to metadata added toan image data element, and the determination of a frame captured withunintended light emission, such as the firing of another flash (byanother photographer or the like) or instantaneously illumination by thelight of a vehicle passing by, may be performed based on the brightnessdistribution of preceding and following frames of consecutive images(sum of brightness signals) or the like. With this determination, afavorable combined image can be obtained even in accidental illuminationconditions such as the firing of a flash by another photographer.

Information regarding the image capture distance may be implemented notonly by an image capture distance (a distance to the in-focus planeduring image capture) which is stored as metadata (for example, Exifdata) but also by, for example, an image capture distance to an in-focussubject obtained using an area extraction technique or the so-calleddistance distribution information (depth map).

The determination of the necessity of correction (weighting) for a framecaptured with the firing of the flash may be performed using areaextraction or distance information based on distance distributioninformation and using the relationship that “flash reach distance(m)=guide number (ISO 100)÷aperture setting value”. If “distanceinformation (m)<flash reach distance (m)” holds, it is determined thatthe amount of flash light is insufficient and that it is necessary tocorrect a weighting coefficient.

If the correction is necessary, a correction value is determined basedon the fact that the amount of flash light is in inverse proportion tothe square of the distance. Also in this case, weighting can beperformed on an area or each pixel using area extraction or distanceinformation based on distance distribution information.

That is, a long-time exposure image is obtained in image combination notonly by performing weighting but also by changing weights according tothe distances within one image according to area extraction based on theimage capture distances or distance information such as distancedistribution information, thereby achieving a more favorablereproduction of the flash effect.

Further, the flash removal described above can also be performed bydetermining whether or not flash light emission has reached and reducingthe weighting on an image with the image capture distance within whichthe flash light emission has reached.

Also in the flash removal, a more favorable image can be obtained byperforming weighting on an area or each pixel using area extraction ordistance information based on distance distribution information.

The flash reach distance has the relationships that “flash reachdistance (m)=guide number (ISO 100)÷aperture setting value” and that theamount of flash light is in inverse proportion to the square of thedistance. Thus, a correction value for each distance can be computedusing the amount of flash light recorded in the metadata.

During correction, weighting may be performed by specifying a guidenumber on the basis of the fact that the amount of flash light is ininverse proportion to the square of the distance. A guide number may bespecified by referring to metadata to display the actual guide number soas to be changed or by entering a desired guide number (which isdifferent from that for the actual flash). Thus, a variety of imagerepresentations are achieved.

11. Exemplary Combination-Mode Processes: Blurring Correction

An exemplary combination-mode process including blur correction will nowbe described.

FIGS. 48A, 48B, and 48C show examples of combined images obtained usinga combination process. Here, subjects in the combined images include amain subject (dynamic subject with motion) that is moving from the upperright to lower left of the screen and a static subject in the backgroundthereof. The term “static subject” as used herein means an object whicha user “does not wish to move” during image capture, rather than anobject appearing in the background or an object that “does not move oris stationary” such as a building or landscape. Static subjects may alsoinclude, for example, people and animals. Static subjects are thereforesubjects whose blurring in a combined image is not desirable.

FIG. 48A shows an image desired by the user, in which the long-timeexposure effect is achieved for a dynamic subject while static subjectsin the background appear stationary.

In the continuous image capture in step F15 shown in FIG. 5, imagecapture is carried out for a certain period of time. Thus, due to theshake of the user's hand or the movement of the subjects, the staticsubjects may blur in a resulting combined image.

FIG. 48B shows blurring of the static subjects in the background due tothe camera shake in a case where, for example, the image captureapparatus 1 is not fixed by a tripod or the like.

FIG. 48C shows blurring of the static subjects due to the arbitrarymovements of the individual subjects (hereinafter referred to as“subject blur”). For example, even if the image capture apparatus 1 isfixed by a tripod or the like, a static subject may unintentionally blurin a combined image due to its movement.

In the following exemplary processes, the influence of such camera shakeor subject blur is corrected by way of example.

First, a camera-shake image is produced when the photographer shakes theimage capture apparatus 1. In this case, static subjects in thebackground similarly blur in a combined image.

Such camera shake is corrected by performing coordinate transformationon each of image data elements in a combination range according tomotion vectors of the static subjects, which are detected in eachcombination-use image data element, and then combining thecombination-use image data elements.

A subject blur is caused by arbitrary movements of individual subjects.That is, static subjects differently blur.

Such a subject blur is addressed by extracting an image area of adynamic subject from each of image data elements in a combination rangeand performing a combination process to combine the image areas of thedynamic subject extracted from the combination-use image data elementswith one combination-use image data element.

First, an exemplary process for correcting camera shake will bedescribed with reference to FIGS. 49 to 52.

FIG. 49 shows a combination-work image 70 in which a combined imageobtained by combining image data elements #5 to #11 in a combinationrange is displayed in the image display area 72 as a preview image. Inthis case, background subjects blur (user's unintentional blurringoccurs) due to the camera shake during image capture.

While the term “camera shake” is used herein for the convenience ofdescription, a frame composition shift in continuous image capture wouldcause similar blurring in a combined image even though the camera is notactually shaken.

Camera-shake correction is performed using motion vectors. FIG. 52 showsan exemplary process including a process for correcting such camerashake. In FIG. 52, steps F220 to F225 are similar to those in FIG. 9. InFIG. 52, steps F270 to F274 are added to the process shown in FIG. 9.

For example, when the combination-work image 70 shown in FIG. 49 isbeing displayed in the adjustment process, as described above, inaddition to the operation of changing a weighting coefficient orchanging a combination range, a user can perform a blur correctionoperation. The blur correction operation can be performed by, forexample, operating a predetermined operation key or selecting a menuitem.

When a blur correction operation performed by the user is detected, theCPU 31 advances the process from step F270 to step F271 and performs,first, a reference image selection process.

The reference image selection process may be a process for selecting areference image data element for correction from among image dataelements #5 to #11 in a combination range.

As a reference image, the top frame is selected from the selection imagelist defined by the combination start position and the combination endposition. The reference image may also be the center frame.Alternatively, the user may be allowed to select a reference image sothat image combination can be performed using the user's favoritebackground condition.

Then, in step F272, the CPU 31 performs a specific point extractionprocess. In this process, the CPU 31 extracts a specific point in eachof the image data elements #5 to #11, and detects amounts ofdisplacement of the specific point on images other than the referenceimage with respect to the specific point on the reference image.

The specific point is a characteristic portion in a static subject, andan amount of displacement corresponds to the motion vector of thespecific point on each image data with respect to a reference imagedata.

The specific point may be extracted by selecting a high-brightnesshigh-contrast image from among a plurality of image data elements (edgedetection) or may be selected by the user using a touch panel, a cursor,or the like. Alternatively, a plurality of coordinates may beautomatically extracted and the user may be allowed to select one ofthem.

For the images other than the reference image, an identical image in thevicinity of (within the range of frame composition blur due to camerashake which is defined by the focal length of the image captureapparatus 1) the position coordinates of the specific point in thereference image or an image close to the reference image (for example,an adjacent frame) is found and extracted.

A frame is selected and checked using a playback image. Thus, theextracted specific point can be checked.

The extracted specific point in the images other than the referenceimage can be modified by operating a touch panel, a cursor, or the like.

Furthermore, instead of merely extracting/selecting a specific pointfrom the reference image, possible specific points may also be extractedfrom the images other than the reference image and a specific pointhaving the smallest movement may be selected. The possible specificpoints may also be arranged in ascending order of movement and the usermay be allowed to select one of them.

Moreover, a correlation arithmetic operation may be executed between thereference image and the images other than the reference image to obtainchanges in the corresponding pixels as motion vectors as changes of thespecific point.

In a case where a specific point is set either automatically or by theuser, a plurality of specific points rather than one specific point maybe selected.

A plurality of points rather than one point may be selected as specificpoints, and the average values of the motion vectors at the points orthe minimum values of scalar may be set as changes of the specificpoints.

The plurality of selected specific points may be weighted to obtainweighted average values of motion vectors.

In step F272, the CPU 31 extracts a specific point for each of the imagedata elements #5 to #11 in the manner described above, and calculatesamounts of displacement of the specific point in non-reference imagedata elements with respect to the reference image. For example, if theimage plane is assumed to be an XY coordinate plane, an amount ofdisplacement may be an amount of shift on the X and Y axes. When aspecific point and amounts of displacement are detected in the mannerdescribed above, in step F273, the CPU 31 performs coordinatetransformation of the images other than the reference image on the basisof the detection results.

For example, if the image data element #5 is used as a reference image,first, in the image data element #6, the amount of displacement of animage at the specific point with respect to the image at the specificpoint in the reference image data element #5 has been determined. Thus,coordinate transformation is performed on the image data element #6 bythe corresponding amount of displacement. Specifically, coordinatetransformation is performed (the image is shifted on the XY coordinates)so that the position (XY coordinate value) of the specific point in theimage data element #6 can match the position of the specific point inthe image data element #5. Coordinate transformation is also performedin a similar manner for the image data elements #7 to #11 so that theposition of the specific point in each of the image data elements #7 to#11 can match the position of the specific point in the reference imagedata element #5.

Then, in step F274, a combination process is performed for each of theimage data elements #5 to #11, and a resulting image is displayed in theimage display area 72 as a preview image.

FIG. 50 shows a combination-work image 70 obtained in this case. Asshown in FIG. 50, a combined image in which the influence of camerashake has been corrected is displayed in the image display area 72.

Camera shake may also influence the main subject with motion. However,the influence of camera shake on the main subject is overcome in thecombination after the coordinate transformation. Thus, a smoothlong-time exposure effect image can be achieved.

Coordinate transformation on image data other than reference image datamay cause a portion where images do not overlap each other. In order toprevent the occurrence of such a portion, a trimming process may beapplied to the image data.

In the example described above, a technique for detecting motion vectorsof a specific point from image data itself. Alternatively, a sensoroutput (for example, a sensor output for detecting an amount anddirection of camera shake detected by the blur detection unit 13) may beincluded in metadata of each image data element so that an amount ofdisplacement of each image data element with respect to a referenceimage data element can be determined using the value of the sensoroutput to perform coordinate transformation.

FIG. 51 shows an example of the multi-flash effect combined imagegenerated in accordance with a weighting coefficient operation performedby the user after a blur correction has been performed.

With the process shown in FIG. 52, the user can obtain a desired imageeffect after performing blur correction in the manner described above.

An exemplary process for correcting subject blur will now be described.

Subject blur is addressed by extracting an image area of a dynamicsubject from each of image data elements in a combination range andperforming a combination process to combine the image areas of thedynamic subject extracted from the combination-use image data elementswith one combination-use image data element.

FIG. 53 shows a combination-work image 70 in which a combined imageobtained by combining image data elements #5 to #11 in a combinationrange is displayed in the image display area 72 as a preview image. Inthis case, background subjects blur due to the subject blur during imagecapture.

Therefore, subject-blur correction is performed. FIG. 54 shows anexemplary process including a process for correcting subject blur. InFIG. 54, steps F220 to F225 are similar to those in FIG. 9. In FIG. 54,steps F280 to F284 are added to the process shown in FIG. 9.

For example, when the combination-work image 70 shown in FIG. 53 isbeing displayed in the adjustment process, as described above, inaddition to the operation of changing a weighting coefficient orchanging a combination range, a user can perform a blur correctionoperation by operating a predetermined operation key or selecting a menuitem.

When a blur correction operation performed by the user is detected, theCPU 31 advances the process from step F280 to step F281 and performs,first, a reference image selection process.

The reference image selection process may be similar to the processdescribed above in the camera-shake correction, and may be a process forselecting a reference image data element for correction from among imagedata elements #5 to #11 in a combination range.

As a reference image, the top frame is selected from the selection imagelist defined by the combination start position and the combination endposition. The reference image may also be the center frame.Alternatively, the user may be allowed to select a reference image sothat image combination can be performed using the user's favoritebackground condition (that is, a frame to be exposed for a long time inthis state without movement of the subject).

Then, in step F282, the CPU 31 performs a motion image selectionprocess.

The user may be allowed to select a subject whose motion is to berepresented using long-time exposure by operating a touch panel, acursor, or the like in, for example, the reference image or any otherimage. In accordance with a user operation, the CPU 31 determines amotion image.

Then, in step F283, the CPU 31 extracts a motion-image area in each ofthe image data elements #5 to #11. That is, the CPU 31 performs aprocess of extracting a coordinate range including the specified motionimage within each of the image data elements #5 to #11.

Area extraction may be performed based on the determination of anoutline as a motion image.

Distance distribution information may also be obtained as metadataduring image capture and a portion having a large difference in distanceinformation may be determined as an outline portion to extract an area.

Alternatively, candidates extractable as areas may be extracted and theuser may be prompted to select one of them.

For images other than an image in which area extraction is first set, anidentical image in the vicinity of (within the range of subject blurthat is defined by the exposure time based on divisional exposure) theposition coordinates of the extracted area in the initially set image oran image close to the initially set image (for example, an adjacentframe) is found and extracted.

A frame is selected and checked using a playback image. Thus, theextracted area can be checked. The extracted area can also be modifiedby operating a touch panel, a cursor, or the like.

In a case where the motion-image area is either set automatically orselected by the user, a plurality of areas rather than one area may beselected.

If a scene where a subject serving as a motion image is moving acrossthe screen (within the angle of view) is captured, continuously capturedimage data elements may include an image data element in which thesubject whose motion is to be represented does not appear. In this case,the user is notified that, for the frame in which no motion-image areais extractable, no extractable candidates are found in the screen bydisplaying the notification on a display screen. Further, due to therapid motion of a subject serving as a motion image, the subject canappear in only one frame. In this case, the user may be notified that anextractable candidate appears in only one image data element.

That is, area extraction may not necessarily be performed for all framesbut may be performed for only one image.

After the area extraction is performed, in step F284, the CPU 31performs a combination process. In this case, the CPU 31 performs aprocess of combining data including the motion-image areas (that is, asubject whose motion is to be represented with the long-time exposureeffect) extracted from each of the image data elements #5 to #11 withthe reference image data element. That is, combination is performedusing only the entire screen of the reference image data element and theareas extracted from the remaining image data elements.

In this process, the background of only the reference image data elementis used to produce a background except for the extracted motion image,and images of a main subject whose motion is to be represented (imagesof the extracted areas) are added to the background of the referenceimage data element.

Then, a preview image obtained by the combination described above isdisplayed in the image display area 72. For example, as shown in FIG.50, a combined image without blurring of the background subjects isobtained. Since the background image (static subjects) is formed onlyusing the reference image data element, no blur occurs.

After blur correction is performed in the manner described above, inaccordance with a user operation, the weighting coefficients of theimage data elements #5 to #11 are changed. Thus, various photographicrepresentations can be achieved.

Note that the lightness or color of a combined image is automaticallyadjusted according to an area used for the combination and thebrightness level thereof, thereby obtaining a combined image havingappropriate lightness/color. For example, if an extracted area iswhitish or bright, high-brightness information of the overall combinedimage will be obtained. A value for performing underexposure correctionis computed based on the brightness information.

A motion image changes in accordance with the motion. For example, whenthe subject is a person or animal, the body size or posture (such as theorientation of the legs or the angle of the neck) differs depending onthe image data elements #5 to #11. It is therefore desirable that areaextraction be performed from each image data element by using a motionimage that is extracted first as a reference and extracting an areaincluding an image similar to the motion image in each of the remainingimage data elements.

The extracted areas may be stored in a memory (for example, the flashROM 33) of the image capture apparatus 1 and may be used for subsequentimage capture or image combination process.

Other area extraction (area division) techniques other than thatdescribed above may also be used, such as using the intensity ofbrightness signals or a threshold. For example, area extraction may beperformed using a brightness signal as a threshold. This can provide acombined image similar to an image of a beam of light moving in a nightscene (for example, the taillight of a vehicle) which is captured usinglong-time exposure.

Next, an exemplary process for correcting both camera shake and subjectblur will be described with reference to FIG. 55.

Here, by way of example, a combination process will be described inwhich an image area of a dynamic subject is extracted from each of imagedata elements in a combination range having continuity in time;coordinate transformation is performed for each of the extracted imageareas of the dynamic subject on the basis of detected motion vectors ofa static subject in the image data elements; and then the image areas ofthe dynamic subject are combined with one reference image data element.

FIG. 55 shows the process of the CPU 31. Steps F290 to F296 show theprocessing involved in a blur correction operation (the remaining steps,namely, steps F220 to F225, are similar to those shown in FIG. 9).

When a blur correction operation performed by the user is detected, theCPU 31 advances the process from step F290 to step F291 and performs,first, a reference image selection process in a manner similar to thatin the examples shown in FIGS. 52 and 54.

Then, in step F292, the CPU 31 performs a motion image selection processin a manner similar to that in the example shown in FIG. 54. Then, instep F293, the CPU 31 extracts a motion-image area in each image dataelement.

In step F294, as in the example shown in FIG. 52, the CPU 31 extracts aspecific point for a static subject and detects amounts of displacementof the specific point. Specifically, an amount of shift of a coordinateposition of the specific point of the static subject in each of imagedata elements other than a reference image data element with respect toa coordinate position of the specific point in the reference image dataelement is detected.

In step F295, the CPU 31 performs coordinate transformation of the areasextracted as the motion image from the individual image data elementsother than the reference image data element using the amounts ofdisplacement.

For example, if the image data element #5 is used as a reference imagedata element, first, coordinate transformation is performed on themotion-image area extracted from the image data element #6 using thedetected amount of displacement of the specific point in the image dataelement #6 with respect to the specific point in the reference imagedata element #5. Coordinate transformation is also performed on themotion-image areas extracted from the image data elements #7 to #11 in asimilar manner.

Then, in step F296, the CPU 31 performs a combination process. In thiscase, the CPU 31 performs a process of combining data of themotion-image areas (that is, a subject whose motion is to be representedwith the long-time exposure effect) that have been extracted from theimage data elements #5 to #11 and that have been coordinate-transformedin step F295 with the reference image data element. That is, combinationis performed using the entire screen of the reference image data elementand the areas extracted from the remaining image data elements.

A preview image obtained by the combination described above is displayedin the image display area 72. For example, as shown in FIG. 50, acombined image without blurring of the background subjects is obtained.Since the background image (static subjects) is formed only by thereference image data element, no influence of camera shake or subjectblur occurs. Further, images of a main subject serving as a motion imageare combined after the coordinate transformation, and no influence ofcamera shake occurs.

Next, another exemplary process for correcting both camera shake andsubject blur will be described with reference to FIG. 56.

In this process, after coordinate transformation based on detectedmotion vectors of a static subject within image data elements isperformed on each of image data elements in a combination range havingcontinuity in time, an image area of a dynamic subject is extracted fromeach of the image data elements and a combination process is performedso that the area images of the dynamic subject extracted from the imagedata elements are combined with one reference image data element.

FIG. 56 shows the process of the CPU 31. Steps F290, F291, F292, F297,F298, F299, and F296 show the processing involved in a blur correctionoperation (the remaining steps, namely, steps F220 to F225, are similarto those shown in FIG. 9).

When a blur correction operation performed by the user is detected, theCPU 31 advances the process from step F290 to step F291 and performs,first, a reference image selection process in a manner similar to thatin the examples shown in FIGS. 52 and 54.

Then, in step F292, the CPU 31 performs a motion image selection processin a manner similar to that in the example shown in FIG. 54.

Then, in step F297, as in the example shown in FIG. 52, the CPU 31extracts a specific point for a static subject and detects amounts ofdisplacement of the specific point. Specifically, an amount of shift ofa coordinate position of the specific point of the static subject ineach of image data elements other than a reference image data elementwith respect to a coordinate position of the specific point in thereference image data element is detected.

Then, in step F298, the CPU 31 performs coordinate transformation on theindividual image data elements other than the reference image dataelement using the amounts of displacement.

For example, if the image data element #5 is used as a reference imagedata element, first, coordinate transformation is performed for theentirety of the image data element #6 using the detected amount ofdisplacement of the specific point in the image data element #6 withrespect to the specific point in the reference image data element #5.Coordinate transformation is also performed for the image data elements#7 to #11 in a similar manner.

After coordinate transformation is performed for the individual imagedata elements other than the reference image data element, in step F299,the CPU 31 extracts a motion-image area from each of the image dataelements #5 to #11.

Then, in step F296, the CPU 31 performs a combination process. In thiscase, the CPU 31 performs a process of combining data of themotion-image areas (that is, a subject whose motion is to be representedwith the long-time exposure effect) extracted from the image dataelements #5 to #11 with the reference image data element. That is,combination is performed using the entire screen of the reference imagedata element and the areas extracted from the remaining image dataelements.

A preview image obtained by the combination described above is displayedin the image display area 72. For example, as shown in FIG. 50, acombined image without blurring of the background subjects is obtained.Since the background image (static subject) is formed only by thereference image data element, no influence of camera shake or subjectblur occurs. Further, images of a main subject serving as a motion imageare extracted from the image data elements #5 to #11 that have beencoordinate-transformed, the influence of camera shake has also beenovercome.

With the exemplary processes described above, a combined image in whichthe influence of camera shake or subject blur has been overcome can beobtained, and a user can more easily obtain his or her desired combinedimage. In particular, users who are inexperienced in image capturetechniques can obtain a combined image or undesired motion of abackground subject during image capture can be corrected.

12. Information Processing Apparatus

In the foregoing embodiment, an image capture and combination processesare performed using the image capture apparatus 1. The combinationprocess may be performed using a device other than the image captureapparatus 1. FIG. 57 shows an information processing apparatus servingas an example of an apparatus configured to execute a combinationprocess, for example, a personal computer 200.

FIG. 57 shows an example structure of a personal computer (hereinafterreferred to as a “PC”) 200.

As shown in FIG. 57, the PC 200 includes a central processing unit (CPU)211, a memory unit 212, a network interface unit 213, a displaycontroller 214, an input device interface unit 215, a hard disk drive(HDD) interface unit 216, a keyboard 217, a mouse 218, an HDD 219, adisplay device 220, a bus 221, an external device interface unit 222,and a memory card interface unit 223.

The CPU 211, which may be a main controller of the PC 200, executesvarious control processes according to a program stored in the memoryunit 212. The CPU 211 is connected to other units via the bus 221.

Each of the devices on the bus 221 has a unique memory address or aninput/output (I/O) address, and the CPU 211 can use the addresses toaccess the devices. An example of the bus 221 may be a peripheralcomponent interconnect (PCI) bus.

The memory unit 212 is configured to include both a volatile memory anda non-volatile memory. The memory unit 212 includes a non-volatilememory such as a ROM for storing a program, a RAM used as a computationwork area or temporary storage of various data, and an electricallyerasable and programmable read only memory (EEPROM).

The memory unit 212 is used to store program code executed by the CPU211 or other information such as identification information unique tothe PC 200, or is used as a buffer area for communication data or as awork area for work data while it is executed.

The network interface unit 213 connects the PC 200 to a network such asthe Internet or a local area network (LAN) according to a predeterminedcommunication protocol such as Ethernet (registered trademark). The CPU211 can communicate with apparatuses connected to the network via thenetwork interface unit 213.

The display controller 214 is a dedicated controller for actuallyprocessing rendering commands issued by the CPU 211. For example, thedisplay controller 214 supports a bitmap rendering functioncorresponding to the Super Video Graphic Array (SVGA) or extendedGraphic Array (XGA) standard. The rendering data processed in thedisplay controller 214 is temporarily written into, for example, a framebuffer (not shown), and is then output to the display device 220. Thedisplay device 220 may be, for example, an organic EL display, a cathoderay tube (CRT) display, a liquid crystal display, or the like.

The input device interface unit 215 is a device for connecting a userinput device including the keyboard 217 and the mouse 218 to a computersystem implemented as the PC 200.

Specifically, a user operation input to the PC 200 is performed usingthe keyboard 217 and the mouse 218, and the operation input informationis supplied to the CPU 211 via the input device interface unit 215.

The HDD interface unit 216 performs an interface process for performingwriting/reading on the HDD 219.

The HDD 219 is an external storage device in which a magnetic diskserving as a storage medium is fixedly mounted, as is common in the art,and has a larger storage capacity and a higher data transfer rate thanother external storage devices. Placing a software program onto the HDD219 in an executable state is referred to as “installing” the programinto the system. In general, the HDD 219 stores program code of anoperating system (OS) executed by the CPU 211, application programs,device drivers, etc., in a non-volatile state.

The programs stored in the HDD 219 are developed in the memory unit 212,for example, when the PC 200 is started or when an application programfor the user layer is started. The CPU 211 performs a process based onthe program developed in the memory unit 212.

The external device interface unit 222 is configured to interface withan external device connected according to a standard such as the USBstandard.

In the present embodiment, examples of external devices may include, forexample, a digital still camera, a video camera, and a video player.

The PC 200 can acquire image data from a digital still camera or thelike through communication via the external device interface unit 222.

The standard supported by the external device interface unit 222 is notlimited to the USB standard but may be any other interface standard suchas the Institute of Electrical and Electronics Engineers (IEEE) 1394.

The memory card interface unit 223 is configured to write/read datainto/from a recording medium 90 such as a memory card.

For example, the recording medium 90, which is used for a digital stillcamera, for example, the image capture apparatus 1 described above, avideo camera, or the like, is placed. Then, image data can be read fromthe recording medium 90.

In the PC 200 having the structure described above, an arithmeticprocessing/control operation based on the software configuration in theCPU 211, namely, software such as the application programs, the OS, andthe device drivers, is performed to execute various operations.

In the present embodiment, the processes of steps ST1 to ST4 describedas the processes in the combination mode shown in FIG. 4, that is, thetarget image selection/acquisition process (ST1), the combiningpreparatory process (ST2), the combination process (ST3), and thecombined-image recording process (ST4), are executable. A program forperforming the processes is installed into, for example, the HDD 219 andis developed in the memory unit 212 when it is started. The CPU 211executes necessary arithmetic processes or control processes accordingto the program developed in the memory unit 212.

Then, in the CPU 211, the program, when started, allows thepre-combination processing unit 52, combination processing unit 53,recording/playback/transmission control unit 54, operation detectionunit 55, display control unit 56, and template management unit 57 shownin FIG. 3 to be configured as the functional blocks.

In other words, the processes described with reference to FIGS. 7, 8,and 9 as well as the processes described with reference to FIGS. 19, 22,36, 38, 40, 44, 52, 54, 55, and 56 are executed in the CPU 211.

Accordingly, the user can use the PC 200 to perform combinationprocesses for obtaining various image effects as described above.

A program for causing the CPU 211 to execute the processes describedabove can be recorded in advance on an HDD serving as a recording mediumincorporated in an apparatus such as the PC 200, or a ROM, a flashmemory, or the like in a microcomputer having a CPU.

Alternatively, the program can be temporarily or permanently stored(recorded) on a removable recording medium such as a flexible disk, acompact disc read only memory (CD-ROM), a magnet-optical (MO) disk, adigital versatile disc (DVD), a Blu-ray disk, a magnetic disk, asemiconductor memory, or a memory card. Such removable recording mediacan be provided as so-called packaged software.

The program may also be downloaded from a download site via a networksuch as a LAN or the Internet as well as installed into a personalcomputer or the like from a removable recording medium.

In the PC 200 having the structure described above, for example, the HDD219 can store various types of image content. For example, image contentcaptured by a user using a digital still camera or a video camera may beacquired and stored into the HDD 219. Thus, the user can enjoy thecaptured image played back using the PC 200.

For example, the external interface unit 222 of the PC 200 may beconnected to the external interface 8 of the image capture apparatus 1so that image data captured using the image capture apparatus 1 can betransferred to the PC 200 and acquired.

The recording medium 90 (memory card) used in the image captureapparatus 1 may also be placed in the memory card interface unit 223 sothat the PC 200 can acquire image data captured using the image captureapparatus 1 from the recording medium 90.

In addition to image content captured by the user, for example, imagecontent played back using an external video player or the like andacquired from the external interface unit 222 or image contentdownloaded from an external server via a network using the networkinterface unit 213 can also be stored in the HDD 219 and played back.

That is, in the PC 200, for example, a plurality of frames of image datahaving continuity in time, which have been captured using a digitalstill camera or a video camera, can be loaded into, for example, the HDD219 for use. A user can use the PC 200 to execute a combination processon the loaded image data in a manner similar to that in the exampledescribed above.

For example, the user performs the combination-mode image capture instep F15 shown in FIG. 5 using the image capture apparatus 1, and thenloads the plurality of captured frames of image data into the PC 200.Then, the user starts software for performing a combination process tocause the CPU 211 to execute the processes of steps ST1 to ST4 describedas the processes in the combination mode shown in FIG. 4. Therefore,various types of image combination can be performed in ahigher-operability environment to create combined images with variousimage effects.

Furthermore, a combination process can be performed not only on imagescaptured by the user himself or herself but also on various types ofimage data (moving-image content) available on the PC 200.

For example, moving-image data loaded into the HDD 219 by any way suchas downloading may be played back, or a DVD drive, a Blu-ray disk drive,or the like, which is not shown in FIG. 57, may be connected so thatimage content recorded on an optical disk such as a DVD or a Blu-raydisk can be played back. In this case, a combination process isperformed on moving-image content recorded on the optical disk, therebygenerating a combined image with a desired image representation such asthe long-time exposure effect, the first-curtain synchronization effect,the second-curtain synchronization effect, or the multi-flash effect.Further, an apparatus incorporating or connected to a televisionbroadcast tuner or the like can generate a combined image of broadcastcontent.

In the present embodiment, a personal computer is used as an informationprocessing apparatus by way of example. Other various informationprocessing apparatuses using image data, such as a mobile phone, apersonal digital assistant (PDA), a game unit, and a video editor, canexecute image combination in a manner similar to that described above.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

What is claimed is:
 1. An image capture apparatus comprising: an imagecapture unit configured to capture an image of a subject to obtain imagedata; an image-capture control unit configured to allow the imagecapture unit to execute an image capture operation of capturing aplurality of frames of image data having continuity in time so that aframe rate of the image capture unit is changed from a first frame rateto a second frame rate, the second frame rate determined in accordancewith a change of a subject brightness level detected during theexecution of the image capture operation; and a combination processingunit configured to perform a combination process using, ascombination-use image data to be combined, the plurality of frames ofimage data having continuity in time captured by the image capture unitso as to generate combined-image data representing a still image,wherein the image-capture control unit performs an exposure adjustmentcontrol operation, comprising extracting at least one non-flash imagefrom the plurality of frames, calculating an average brightness valuefor each of the extracted non-flash images, classifying the extractednon-flash images into a plurality of groups based on the averagebrightness values, designating one of the groups as a representativegroup, calculating an average value of average brightness values for therepresentative group, calculating a correction coefficient for each ofthe extracted non-flash images by comparing the average brightness valueof the extracted non-flash image to the average value, and for each ofthe extracted non-flash images, multiplying all of the pixels of theextracted non-flash image by the correction coefficient.
 2. The imagecapture apparatus according to claim 1, wherein the image capture unitincludes an optical system including an aperture mechanism, and whereinthe image-capture control unit performs exposure adjustment controlduring the execution of the image capture operation by controlling theaperture mechanism and by variably controlling the frame rate.
 3. Theimage capture apparatus according to claim 1, wherein the image captureunit includes an optical system including a light intensity filtermechanism, and wherein the image-capture control unit performs exposureadjustment control during the execution of the image capture operationby controlling the light intensity filter mechanism and variablycontrolling the frame rate.
 4. The image capture apparatus according toclaim 1, wherein the image capture unit includes an image-capture signalprocessing system including a variable gain circuit, and wherein theimage-capture control unit performs exposure adjustment control duringthe execution of the image capture operation by controlling the variablegain circuit in response to a captured image signal and variablycontrolling the frame rate.
 5. The image capture apparatus according toclaim 1, further comprising a recording unit configured to record theplurality of frames of image data having continuity in time captured bythe image capture unit onto a recording medium as a sequence of imagedata used for a combination process.
 6. The image capture apparatusaccording to claim 5, wherein the combination processing unit performs acombination process using the plurality of frames of image data havingcontinuity in time recorded onto the recording medium so as to generatecombined-image data representing a still image.
 7. The image captureapparatus according to claim 1, further comprising an operationdetection unit configured to detect operation input information used fora combination process, wherein the combination processing unit performsa combination process on combination-use image data of frames in a rangeon a time axis specified by the operation input information among thecombination-use image data having continuity in time so as to generatecombined-image data representing a still image.
 8. The image captureapparatus according to claim 1, further comprising an operationdetection unit configured to detect operation input information used fora combination process, wherein the combination processing unit performsa combination process on combination-use image data of each of aplurality of frames using a weighting coefficient specified by theoperation input information so as to generate combined-image datarepresenting a still image.
 9. The image capture apparatus according toclaim 1, wherein the combination processing unit performs a combinationprocess on combination-use image data of each of a plurality of framesusing weighted averages so as to generate combined-image datarepresenting a still image.
 10. The image capture apparatus according toclaim 1, wherein the combination processing unit performs a combinationprocess on combination-use image data of each of a plurality of framesby assigning a weighting coefficient, the weighting coefficient beingdetermined as an inverse of a ratio of lengths of frame periods of thecombination-use image data in the image capture operation, so as togenerate combined-image data representing a still image.
 11. The imagecapture apparatus according to claim 1, wherein the combinationprocessing unit performs a combination process using the combination-useimage data and interpolated image data, the interpolated image databeing generated by an interpolation process using the combination-useimage data, so as to generate combined-image data representing a stillimage.
 12. The image capture apparatus according to claim 1, furthercomprising a display control unit configured to output thecombined-image data generated by the combination processing unit asimage data used for display.
 13. The image capture apparatus accordingto claim 5, wherein the recording unit records the combined-image datagenerated by the combination processing unit onto a recording medium.14. The image capture apparatus according to claim 1, further comprisinga sending unit configured to send the combined-image data generated bythe combination processing unit to an external device.
 15. An imagecapture method comprising the steps of: executing an image captureoperation of capturing a plurality of frames of image data havingcontinuity in time; changing a frame rate of image capture from a firstframe rate to a second frame rate, the second frame rate determined inaccordance with a change of a subject brightness level detected duringthe execution of the image capture operation; and performing acombination process using, as combination-use image data to be combined,the plurality of frames of image data having continuity in time so as togenerate combined-image data representing a still image, wherein anexposure adjustment control operation is performed on the plurality offrames, the exposure adjustment control operation comprising extractingat least one non-flash image from the plurality of frames, calculatingan average brightness value for each of the extracted non-flash images,classifying the extracted non-flash images into a plurality of groupsbased on the average brightness values, designating one of the groups asa representative group, calculating an average value of averagebrightness values for the representative group, calculating a correctioncoefficient for each of the extracted non-flash images by comparing theaverage brightness value of the extracted non-flash image to the averagevalue, and for each of the extracted non-flash images, multiplying allof the pixels of the extracted non-flash image by the correctioncoefficient.